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2  Introduction 

This  project  is  aimed  at  improving  the  state  of  the  art  of  image-guided  and  minimally 
invasive  procedures  by  developing  a  new  generation  of  clinical  techniques  along  with  the 
computer-based  hardware  and  software  needed  for  their  implementation.  The  focus  of  the 
project  is  to  develop  physician  assist  systems  incorporating  robotics,  tracking,  and 
visualization  to  improve  the  precision  of  instrument  placement  and  manipulation  in 
minimally  invasive  procedures.  The  project  is  led  by  the  Imaging  Sciences  and 
Information  Systems  (ISIS)  Center  of  the  Department  of  Radiology  at  Georgetown 
University.  Project  collaborators  include  the  Department  of  Radiation  Medicine  at 
Georgetown,  the  Urology  Robotics  Group  at  Johns  Hopkins  Medical  Institutions,  the 
NSF  sponsored  Engineering  Research  Center  for  Computer  Integrated  Surgical  Systems 
and  Technology  at  Johns  Hopkins  University,  and  the  Engineering  School  at  the  Catholic 
University  of  America,  as  well  as  several  commercial  collaborators.  Commercial 
collaborators  include  Kitware  Inc.,  Traxtal  Inc.,  Accuray  Inc.,  and  Siemens  Medical 
Systems. 

3  Report  Body 

This  section  describes  the  research  accomplishments  associated  with  each  task  in  the 
statement  of  work.  A  brief  overview  will  be  given  and  the  reader  will  be  referred  to  the 
published  papers  in  the  appendix  for  more  details.  This  is  an  annual  report  and  includes 
research  performed  from  December  22,  2004  to  December  20,  2005.  The  award  number 
is  W8 1 X WH-04- 1-0078. 

3.1  Task  1:  Spinal  Robotics  for  Precision  Placement 

This  work  has  focused  on  two  tasks:  1)  continuation  of  the  second  phase  of  an  FDA 
approved  clinical  trial  for  robotically  assisted  spinal  nerve  blocks;  and  2)  continued  work 
in  developing  tracking  algorithms  for  robotically  assisted  lung  biopsy.  As  part  of  this 
work,  Dr.  Cleary  recently  completed  an  article  on  interventional  robotic  systems.  This 
article  is  provided  in  the  appendix  [Cleary  2006]. 

As  detailed  in  the  last  annual  report,  the  initial  clinical  trial  of  20  patients  using  the 
needle  driver  robot  for  spinal  blocks  was  conducted  at  Georgetown  University  from 
August  2002  to  December  2002.  A  picture  of  the  robot  is  shown  in  Figure  1  and  a  picture 
of  the  first  patient  is  shown  in  Figure  2.  The  work  is  in  collaboration  with  the  Urology 
Robotics  Program  at  Johns  Hopkins  Medical  Institutions  (Dan  Stoianovici,  PhD).  After 
some  delays  due  to  other  pressing  projects,  we  have  started  recruiting  patients  for  this 
clinical  trial  and  recently  did  four  additional  cases  (we  are  approved  for  up  to  80 
additional  patients  but  plan  to  collect  20  more  patients  at  this  point). 
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Figure  1:  Robotic  device  showing  mechanical  arm  and  joystick  control 

(courtesy  of  Dan  Stoianovici,  PhD,  Johns  Hopkins  Urology  Robotics) 


Figure  2:  Clinical  trial  of  robotic  device  for  nerve  and  facet  blocks  at  Georgetown 
University  (interventional  radiologist  is  Vance  Watson,  MD) 

The  robotically  assisted  lung  biopsy  work  continues  to  evolve.  The  initial  work  on  this 
project  was  done  by  Sheng  Xu,  PhD,  who  recently  received  his  PhD  from  Johns  Hopkins 
University.  This  work  was  collaboration  with  the  NSF  sponsored  Engineering  Research 
Center  for  Computer  Integrated  Surgical  Systems  and  Technology  at  Johns  Hopkins 
University  (Russell  Taylor,  PhD).  The  work  is  being  published  and  presented  at  the  SPIE 
Medical  Imaging  2006  conference  this  month  (February  2006)  and  the  abstract  is 
reproduced  in  the  next  paragraph.  Copies  of  this  paper  [Xu  2006]  and  the  others  are  in  the 
appendix. 

Lung  biopsy  is  a  common  interventional  radiology  procedure.  One  of  the  difficulties  in 
performing  the  lung  biopsy  is  that  lesions  move  with  respiration.  This  paper  presents  a 
new  robotically  assisted  lung  biopsy  system  for  CT  fluoroscopy  that  can  automatically 
compensate  for  the  respiratory  motion  during  the  intervention.  The  system  consists  of  a 
needle  placement  robot  to  hold  the  needle  on  the  CT  scan  plane,  a  radiolucent  Z-frame 
for  registration  of  the  CT  and  robot  coordinate  systems,  and  a  frame  grabber  to  obtain  the 
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CT  fluoroscopy  image  in  real-time  (Figure  3).  The  CT  fluoroscopy  images  are  used  to 
noninvasively  track  the  motion  of  a  pulmonary  lesion  in  real-time.  The  position  of  the 
lesion  in  the  images  is  automatically  determined  by  the  image  processing  software  and 
the  motion  of  the  robot  is  controlled  to  compensate  for  the  lesion  motion.  The  system  was 
validated  under  CT  fluoroscopy  using  a  respiratory  motion  simulator.  A  swine  study  was 
also  done  to  show  the  feasibility  of  the  technique  in  a  respiring  animal. 


Passive  Arm 


Cartesian  Bridge 


Image  Processing  Computer 

Figure  3:  System  components  for  robotically  assisted  lung  biopsy  using  CT  fluoroscopy 


In  addition  to  this  work  on  developing  tracking  algorithms,  we  also  began  to  investigate 
registration  methods  for  merging  CT  fluoroscopy  with  the  pre-procedure  CT  data  set 
[Yaniv  2006a].  The  details  are  given  in  the  paper  in  the  appendix  and  the  abstract  and 
some  figures  are  reproduced  below. 


Lung  cancer  screening  for  early  diagnosis  is  a  clinically  important  problem.  One 
screening  method  is  to  test  tissue  samples  obtained  from  CT-fluoroscopy  (CTF)  guided 
lung  biopsy.  CTF  provides  real-time  imaging;  however  on  most  machines  the  view  is 
limited  to  a  single  slice.  Mentally  reconstructing  the  direction  of  the  needle  when  it 
is  not  in  the  imaging  plane  is  a  difficult  task.  We  are  currently  developing  3D 
visualization  software  that  will  augment  the  physician's  ability  to  perform  this  task.  At 
the  beginning  of  the  procedure  a  CT  scan  is  acquired  at  breath-hold.  The  physician  then 
specifies  an  entry  point  and  a  target  point  on  the  CT.  As  the  procedure  advances  the 
physician  acquires  a  CTF  image,  at  breath-hold;  the  system  then  registers  the  current 
setup  to  the  CT  scan,  enabling  comparison  between  the  plan  and  current  situation.  As  the 
CT  and  CTF  data  are  acquired  at  different  breath-holds  we  expect  them  to  exhibit 
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displacements  of  up  to  4mm.  To  assess  the  performance  of  different  registration 
algorithms  for  CTF/CT  registration  we  propose  to  use  simulated  CTF  images.  These 
images  are  created  by  deforming  the  original  CT  volume  and  extracting  a  slice  from  it. 
Realistic  deformation  of  the  CT  volume  is  achieved  by  using  positional  information  from 
electromagnetically  tracked  fiducials,  acquired  throughout  the  respiratory  cycle.  To 
estimate  the  dense  displacement  field  underlying  the  sparse  displacement  field  provided 
by  the  fiducials  we  use  radial  basis  function  interpolation.  Finally,  we  evaluated  Thirion's 
"demons"  algorithm,  as  implemented  in  ITK,  for  the  task  of  slice-to-volume  registration. 
We  found  it  to  be  unsuitable  for  this  task,  as  in  most  cases  the  recovered  displacements 
were  less  than  50%  of  the  original  ones.  As  part  of  this  task,  we  developed  an  algorithm 
for  segmentation  of  the  lung  region  as  shown  in  Figure  4. 


Figure  4:  Segmentation  of  lung  region  from  CT  data  set  (left  side)  and  volume 
rendering  of  segmented  region  (right  side) 


3.2  Task  2:  Organ  Tracking  using  Electromagnetic 
Position  Sensing 

The  goal  of  this  task  is  to  investigate  the  use  of  electromagnetic  tracking  for  precisely 
locating  internal  organs  such  as  the  liver  during  interventional  procedures.  This  is  an 
ongoing  collaboration  with  Northern  Digital  (Waterloo,  Canada)  and  Traxtal 
Technologies  (Bellaire,  Texas).  Northern  Digital  has  developed  the  AURORA™ 
electromagnetic  tracking  system,  which  enables  instruments  that  are  fitted  with  a  sensing 
coil  to  be  tracked  and  overlaid  on  an  image  of  the  anatomy.  Our  research  group  at 
Georgetown  served  as  a  beta  test  site  and  was  one  of  the  first  research  groups  worldwide 
to  receive  this  equipment  (Figure  5).  The  system  has  recently  been  released  as  a 
commercial  product  and  we  are  actively  pursuing  several  applications  with  this 
technology. 


Page  8 


Periscopic  Spine  Surgery 


Annual  Report:  22  Dec  04  -  21  Dec  05 


Figure  5:  AURORA™  sensors,  electromagnetic  tracking  system  components, 

and  measurement  volume 

The  left  picture  shows  (from  left  to  right)  the  control  unit,  sensor  interface  device,  and  electromagnetic 
field  generator.  The  middle  picture  shows  the  sensor  coils  along  with  the  electrical  wires  protruding  from 
the  coil,  compared  to  a  match.  The  right  picture  shows  the  measurement  volume  in  mm  relative  to  the 
location  of  the  field  generator.  (Photos  courtesy  of  Northern  Digital,  Inc.) 


In  the  past  year,  we  have  focused  on  developing  this  technology  for  liver  procedures  in 
the  CT  suite.  As  part  of  this  research,  we  have  developed  an  image-guided  surgery 
system  incorporating  electromagnetic  tracking.  Figure  6  shows  the  tracking  system  as 
recently  used  in  an  approved  swine  study  in  the  CT  suite.  Electromagnetic  tracking  of 
instruments  in  the  interventional  suite  enables  image  overlay  of  preoperative  CT  scans 
and  provides  multimodality  guidance  while  decreasing  x-ray  dose. 


Figure  6:  Swine  study  in  CT  suite  with  tracking  of  instruments  and  virtual  overlay 
on  monitor  (interventional  radiologist  is  Filip  Banovac,  MD) 
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In  addition  to  this  study  in  the  CT  suite,  we  carried  out  extensive  experiments  to 
characterize  the  performance  of  electromagnetic  tracking  in  the  fluoroscopy  procedure 
room  as  well  [Yaniv  2006b],  The  abstract  and  a  figure  are  reproduced  below. 

Tracking  organ  motion  due  to  respiration  is  important  to  enable  precise  interventions  in 
the  regions  of  the  abdomen  and  thorax.  Respiratory  induced  motion  in  these  regions  may 
limit  the  accuracy  of  interventions  which  do  not  employ  some  type  of  tracking.  One 
method  of  tracking  organ  motion  is  to  use  a  predictive  model  based  on  external  tracking 
that  is  correlated  to  internal  motion.  This  approach  depends  on  the  accuracy  of  the  model 
used  for  correlating  the  two  motions.  Ideally,  one  would  track  the  internal  motion 
directly.  We  are  investigating  the  use  of  electromagnetically  tracked  fiducials  to  enable 
real-time  tracking  of  internal  organ  motion.  To  investigate  the  in-vivo  accuracy  of  this 
approach  we  propose  to  use  stereo-fluoroscopy.  In  this  paper  we  show  that  stereo¬ 
fluoroscopy  is  accurate  enough  to  serve  as  a  validation  method,  displaying  sub- 
millimetric  accuracy  (maximal  error  of  0.66mm).  We  study  the  effect  of  the  bi-plane 
fluoroscopes  on  the  electromagnetic  systems'  accuracy,  and  show  that  placing  the  bi¬ 
plane  fluoroscopes  in  a  typical  intra-operative  setup  has  a  negligible  effect  on  the 
tracking  accuracy  (maximal  error  of  1.4mm).  Finally,  we  compare  the  results  of  stereo¬ 
fluoroscopy  tracking  and  electromagnetic  tracking  of  needles  in  an  animal  study,  showing 
a  mean  (std)  difference  of  1.4  (1.5)  mm  between  modalities.  These  results  show  that 
stereo-fluoroscopy  can  be  used  in  conjunction  with  electromagnetic  tracking 
with  minimal  effect,  and  that  the  electromagnetic  system  is  accurate  enough  for  motion 
tracking  of  internal  organs.  A  picture  of  the  test  setup  is  shown  in  Figure  7. 


Figure  7:  Electromagnetic  tracking  study  in  biplane  fluoroscopy  suite  showing  both 
fluoroscopes,  dummy  torso  and  moving  liver,  electromagnetically  tracked  needles 
embedded  in  liver,  and  electromagnetic  field  generator 
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3.3  Task  3:  Tracking  for  Radiation  Treatment 

Organs  in  the  abdomen  and  thorax  move  and  deform  during  respiration,  and  these 
changes  place  a  severe  limitation  on  the  effectiveness  of  radiation  medicine  in  the  lungs, 
liver,  and  pancreas.  Georgetown  University  Medical  Center  is  an  emerging  leader  in  a 
new  form  of  radiosurgery  for  these  types  of  tumors.  These  treatments  are  enabled  by  two 
key  technologies:  (1)  the  CyberKnife,  a  precision  radiosurgery  machine  based  around  a 
compact  linac  and  a  robotic  positioning  arm,  and  (2)  Synchrony,  a  system  for  optically 
tracking  respiratory  motion  and  steering  the  radiation  beam  to  follow  a  moving  tumor. 
This  research  is  being  carried  out  in  conjunction  with  the  Radiation  Medicine  Department 
at  Medstar  Georgetown  University  Hospital. 


Figure  8:  Photograph  of  CyberKnife  Stereotactic  Radiosurgery  System 
(Accuray,  Sunnyvale,  CA) 


As  part  of  this  task,  we  performed  an  accuracy  evaluation  of  electromagnetic  tracking  in 
the  CyberKnife  suite  [Wilson  2006].  Electromagnetic  trackers  have  found  inroads  into 
medical  applications  as  a  tool  for  navigation  in  recent  years.  Their  susceptibility  to 
interference  from  both  electromagnetic  and  ferromagnetic  sources  has  prompted  several 
accuracy  assessment  studies  in  past  years.  To  the  best  of  our  knowledge,  this  is  the  first 
accuracy  study  conducted  to  characterize  measurement  accuracy  of  an  NDI  AURORA 
electromagnetic  tracker  within  a  CyberKnife  radiosurgery  suite.  This  paper  uses  a  data 
collection  protocol  to  collect  uniformly  distributed  data  points  within  a  subset  of  the 
AURORA  measurement  volume  in  a  CyberKnife  suite.  The  key  aim  of  the  study  is  to 
determine  the  extent  to  which  large  metal  components  of  the  CyberKnife  stereotactic 
radiosurgery  device  and  robot  mount  contribute  to  overall  system  performance  for  the 
AURORA  electromagnetic  device.  A  secondary  goal  of  the  work  is  to  determine  the 
variation  in  accuracy  and  device  behavior  with  the  presence  of  ionizing  radiation  when 
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the  LINAC  is  turned  on.  The  test  setup  in  shown  in  Figure  9  and  the  details  of  the 
experiments  can  be  found  in  the  paper  in  the  appendix  [Wilson  2006]. 
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Figure  9:  Test  setup  within  the  CyberKnife  suite 
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3.4  Task  4:  Rehabilitation  Robotics 

The  primary  goal  of  this  effort  is  to  develop  robotic  systems  for  assessment  and  physical 
therapy.  The  secondary  goal  is  to  expand  this  capability  to  the  Internet.  We  currently 
have  several  projects  in  rehabilitation  robotics  spanning  three  robotic  testbeds  described 
briefly  below.  We  will  continue  to  expand  our  activities  in  this  area  through  ongoing 
collaborations  with  the  National  Rehabilitation  Hospital,  the  University  of  Maryland,  the 
Center  of  Excellence  for  Remote  and  Medically  Under  Served  Areas  (CERMUSA),  and 
Interactive  Motion  Technologies,  Inc.  As  some  of  these  devices  may  be  controlled 
through  the  Internet,  a  systematic  study  of  information  assurance  and  vulnerabilities  will 
be  an  integral  part  of  this  effort.  Details  of  this  work  are  provided  in  the  three  papers  in 
the  appendix:  [Carignan  2005],  [Roderick  2005],  and  [Tang  2005], 

3.4.1  lnMotion2  Robot 

We  are  currently  using  the  IM2  testbed  for  two  projects:  a  neuroscience  clinical  trial  and 
telerobotic  rehabilitation  over  the  Internet.  The  first  project  is  to  test  sensorimotor 
adaptation  of  subjects  when  exposed  to  kinematic  and  force  distortions  (Figure  10).  The 
eventual  goal  of  this  project  is  to  develop  therapy  for  Parkinsonian  patients.  The  trial  is 
now  complete  and  we  are  currently  processing  the  data.  The  second  project  is  to  develop 
cooperative,  functional  rehabilitation  for  populations  with  neurological  deficits  using 
virtual  reality  technologies.  We  have  developed  several  tasks,  the  most  recent  is  a 
cooperative  tandem  canoe  task  that  can  be  performed  over  the  Internet  between  patient 
and  therapist  (Figure  11). 


Figure  10:  Operator  interface  being  used  for  Figure  11:  Cooperative  tandem  canoe  task 
sensorimotor  clinical  trial. | using  InMotion2  Robot. 


3.4.2  MGA  Arm  Exoskeleton 

We  are  currently  building  an  upper  arm  robotic  exoskeleton  for  shoulder  rehabilitation. 
This  project  is  a  joint  venture  between  CAIMR  and  the  Space  Systems  Laboratory  (SSL) 
at  the  University  of  Maryland.  The  mechanical  hardware  was  designed  and  built  by  the 
SSL  robotics  group,  and  CAIMR  is  developing  the  electronics  and  control  system.  The 
hardware  for  the  exoskeleton  has  been  built  (Figure  12  and  Figure  13),  and  the  system  is 
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currently  undergoing  electronic  integration.  We  expect  operation  to  begin  in  the  summer 


of  2006. 


Figure  12:  Exoskeleton  prototype  being  test  Figure  13:  Close-up  of  the  exoskeleton 
fitted  on  human  subject. shoulder  rotation  assembly. 


3.4.3  Keiser  Arm  Curl  Machine 

We  are  developing  an  actively  controlled  exercise  machine  for  elbow  rehabilitation.  A 
Keiser  Arm  Curl  250  was  retrofitted  with  a  motor  and  lead  screw  assembly  as  shown  in 
Figure  14.  Resistance  laws  were  programmed  on  a  PC  and  an  admittance  controller 
outputs  commands  to  the  motor.  Preliminary  results  for  simulating  free  weight  and 
viscoelastic  resistances  were  encouraging  (Figure  15).  Future  clinical  work  will  focus  on 
designing  resistance  laws  tailored  for  individual  rehabilitation  of  the  hand/elbow 
following  surgery. 


40 

35 


Figure  14:  Keiser  Arm  Curl  250 
retrofitted  with  SmartMotor  actuator. 


Flexion  Angle  (deg) 

Figure  15:  Experimental  results  for  simulating 
free  weight  repetition. 
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3.5  Task  5:  Interventional  Suite  of  the  Future 

The  goal  of  this  task  is  to  develop  the  integrated  interventional  suite  of  the  future.  This 
task  is  a  follow-up  to  the  OR2020  workshop  (Operating  Room  of  the  Future)  organized 
by  Dr.  Cleary  in  March  2004.  During  the  past  year,  we  have  worked  with  Georgetown 
University  Hospital  and  MedStar  Health  to  issue  a  purchase  order  for  a  new 
interventional  suite  in  the  Department  of  Radiology.  This  suite  is  based  on  a  Siemens 
angiography  system  with  a  flat  panel  detector.  The  flat  panel  provides  greater  dynamic 
range  which  should  enable  better  visualization  of  soft  tissue  during  image-guided 
interventions.  We  are  currently  working  out  the  details  of  a  research  agreement  with 
Siemens  to  integrate  the  electromagnetic  tracking  technology  described  above  into  this 
new  interventional  suite. 

In  related  work,  we  have  been  developing  the  software  architecture  for  an  open  source 
toolkit  for  image-guided  surgery  (IGSTK).  This  software  project  is  jointly  funded  by  the 
National  Institute  for  Biomedical  Imaging  and  Bioengineering  at  NIH.  The  open  source 
movement  is  growing  in  popularity  since  it  allows  researcher  to  easily  share  software  and 
avoids  duplication  of  effort.  Since  software  is  increasingly  important  in  medical  research, 
open  source  can  greatly  improve  the  productivity  of  research  groups. 

The  IGSTK  project  will  be  demonstrated  this  month  (February  2006)  at  the  SPIE  Medical 
Imaging  Conference  in  San  Diego.  The  Image-Guided  Surgery  Toolkit  (IGSTK)  is  an 
open  source  software  toolkit  based  on  ITK,  VTK,  and  FLTK,  and  uses  the  cross-platform 
tools  CMAKE  and  DART  to  support  common  operating  systems  such  as  Linux, 
Windows,  and  MacOS.  IGSTK  integrates  the  basic  components  needed  in  surgical 
guidance  applications  and  provides  a  common  platform  for  fast  prototyping  and 
development  of  robust  image-guided  applications.  The  paper  in  the  appendix  gives  an 
overview  of  the  IGSTK  framework  and  current  status  of  development  followed  by  an 
example  needle  biopsy  application  using  this  toolkit  [Cheng  2006]. 
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Figure  16:  IGSTK  open  source  toolkit  screen  shot 

3.6  Task  6:  Nano-Technology  Medical  Sensors 

This  task  is  to  develop  a  nano-sensor  that  can  be  used  on  a  probe  tip  to  assess  cell  death 
in  applications  such  as  targeted  drug  or  radiation  therapy.  This  is  a  new  collaboration 
with  the  Physics  Department  at  Georgetown  and  Professor  John  Currie,  who  is  an  expert 
in  micro-fabrication  techniques.  Unfortunately,  due  to  time  constraints  on  Dr.  Currie  and 
several  other  pressing  projects  in  his  laboratory,  this  task  has  not  made  progress  during 
the  past  year.  However,  we  have  had  recent  renewed  discussions  with  Dr.  Currie  and  are 
optimistic  about  moving  this  project  forward  in  2006. 

3.7  Task  7:  Grid  Computing  &  Electronic  Patient  Record 

This  project  is  a  medical  informatics  task  to  look  at  the  electronic  patient  record  and  the 
use  of  emerging  technologies  such  as  grid  computing  to  provide  a  seamless  repository  of 
clinical  records.  During  the  past  year,  we  have  begun  this  task  by  establishing  a  workflow 
project  in  interventional  radiology  to  track  patient  cases  during  spinal  nerve  blocks.  The 
idea  is  that  workflow  analysis  can  show  us  where  the  bottlenecks  are  in  the  system  and 
what  information  is  needed  at  what  time  and  in  what  place.  Our  initial  workflow  results 
will  be  reported  at  SPIE  Medical  Imaging  2006  [Siddoway  2006]  and  are  summarized 
here. 

Workflow  management  has  the  potential  to  dramatically  improve  the  efficiency  and 
clinical  outcomes  of  medical  procedures.  In  this  study,  we  recorded  the  workflow  for 
nerve  block  and  facet  block  procedures  in  the  interventional  radiology  suite  at 
Georgetown  University  Hospital  in  Washington,  DC,  USA.  We  employed  a  custom 
client  /  server  software  architecture  developed  by  the  Innovation  Center  for  Computer 
Assisted  Surgery  (ICCAS)  at  the  University  of  Leipzig,  Germany.  This  software  runs  in 
an  internet  browser,  and  allows  the  user  to  record  the  actions  taken  by  the  physician 
during  a  procedure  as  shown  in  Figure  17.  The  data  recorded  during  the  procedure  is 
stored  as  an  XML  document,  which  can  then  be  further  processed.  We  have  successfully 
gathered  data  on  a  number  if  cases  using  a  tablet  PC  with  a  touch  screen,  and  these 
preliminary  results  show  the  feasibility  of  using  this  software  in  an  interventional 
radiology  setting.  We  are  currently  accruing  additional  cases  and  when  more  data  has 
been  collected  we  will  analyze  the  workflow  of  these  procedures  to  look  for 
inefficiencies  and  potential  improvements. 
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Figure  17:  Interface  screen  for  recording  workflow  during  spinal  nerve  blocks 


4  Key  Research  Outcomes 

This  section  provides  a  bulleted  list  of  key  research  accomplishments  during  the  entire 
project: 

•  Investigated  methods  for  slice  to  volume  registration  of  CT  fluoroscopy  images  to 
provide  image  guidance  for  lung  biopsy 

•  Completed  an  approved  swine  study  in  the  CT  procedure  room  demonstrating  the 
feasibility  of  electromagnetic  position  sensing  for  image  guidance 

•  Demonstrated  an  accuracy  of  1.4  mm  with  a  standard  deviation  of  1.5  mm  for 
electromagnetic  position  sensing  in  the  interventional  suite  using  biplane 
fluoroscopy 

•  Completed  an  initial  accuracy  study  of  electromagnetic  position  sensing  in  the 
CyberRnife  suite  and  showed  that  relatively  good  accuracy  could  be  obtained 
even  with  the  LINAC  (radiation  beam)  turned  on 


Page  17 


Periscopic  Spine  Surgery 


Annual  Report:  22  Dec  04  -  21  Dec  05 


•  Developed  the  software  architecture  and  software  implementation  for  an  open 
source  image-guided  surgery  toolkit 

•  Completed  an  initial  study  of  workflow  in  the  interventional  suite  for  spinal  nerve 
blocks 

5  Reportable  Outcomes 

This  section  provides  a  list  of  reportable  outcomes.  The  major  product  of  this  year  is  the 
list  of  papers  given  in  Section  7  (References).  Copies  of  these  documents  are  provided  in 
the  appendix. 

In  addition,  several  proposals  to  the  National  Institutes  of  Health  were  submitted  based 
on  this  work.  Several  small  business  grants  were  received  with  company  partners, 
including  one  on  electromagnetic  tracking  for  the  CyberKnife  suite. 

A  master’s  student  from  Catholic  University  (Emmanuel  Wilson)  was  supported  for  the 
electromagnetic  tracking  work  in  CT  and  the  CyberKnife  suite.  A  PhD  student  from 
Johns  Hopkins  University  (Sheng  Xu)  was  supported  to  assist  in  software  development 
for  the  robotic  needle  driver.  The  research  group  at  Georgetown  continued  to  take  a  lead 
in  the  Washington  Area  Computer  Aided  Surgery  Society  (www.washcas.org),  which 
was  formed  in  2000  to  promote  research  in  the  field. 


6  Conclusions 

The  Periscopic  Spine  Surgery  project  has  continued  to  lay  the  ground  work  for 
developing  the  physician  assist  systems  of  the  future.  These  systems  will  incorporate 
robotics,  tracking,  and  visualization  to  improve  the  precision  of  instrument  placement  and 
manipulation  in  minimally  invasive  procedures.  We  have  begun  to  show  the  feasibility  of 
electromagnetic  position  sensing  and  image  overlay  in  the  clinical  environment  and  plan 
to  apply  for  FDA  approval  soon.  We  have  reached  an  agreement  to  install  a  new 
angiography  suite  at  the  hospital  incorporating  a  flat  panel  detector  and  many  of  our 
future  activities  will  be  focused  here.  We  have  continued  our  work  on  respiratory  motion 
compensation  as  a  theme  which  cuts  across  many  of  our  efforts.  This  project  has  enabled 
the  Georgetown  team  to  become  a  world  leader  in  the  emerging  fields  of  computer  aided 
surgery  and  medical  robotics.  Our  goal  will  continue  to  be  to  develop  systems  to  add  the 
physician  in  these  demanding  minimally  invasive  procedures  with  the  ultimate  aim  of 
improving  patient  care. 
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8  Appendices  (Papers) 


Copies  of  the  eight  papers  published  during  this  report  period  are  reproduced  in  this 
section. 

8.1  Carignan  2005:  Design  of  an  Exoskeleton  ... 

Reprint  begins  on  the  next  page  and  is  eight  pages. 
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Abstract—  The  evolution  of  an  arm  exoskeleton  design  for 
treating  shoulder  pathology  is  examined.  Tradeoffs  between 
various  kinematics  configurations  are  explored,  and  a  device 
with  five  active  degrees  of  freedom  is  proposed.  Two  rapid- 
prototype  designs  were  built  and  fitted  to  several  subjects  to  verify 
the  kinematic  design  and  determine  passive  link  adjustments. 
Control  modes  are  developed  for  exercise  therapy  and  functional 
rehabilitation,  and  a  distributed  software  architecture  that  in¬ 
corporates  computer  safety  monitoring  is  described.  Although 
intended  primarily  for  therapy,  the  exoskeleton  will  also  be  used 
to  monitor  progress  in  strength,  range  of  motion,  and  functional 
task  performance. 


I.  Introduction 

The  development  of  robotic  exoskeletons  for  physical  ther¬ 
apy  is  relatively  recent  in  the  field  of  robotics.  Powered 
orthotic  devices  have  been  in  use  for  over  a  decade,  but  the 
focus  of  these  mechanisms  has  been  assistive  rather  than  reha¬ 
bilitative.  The  primary  role  of  exoskeletons  thus  far  has  been 
as  a  haptic  device  for  virtual  reality  (VR)  applications.  Prime 
examples  of  these  devices  include  the  portable,  back-mounted 
EXOS  Force  ArmMaster  and  the  floor-mounted  FREFLEX 
Exo skeleton  [4]. 

When  designing  portable  exoskeletons,  the  classic  tradeoff 
between  power  and  weight  always  emerges.  VR  exoskeletons 
are  almost  always  motor-driven  in  order  to  attain  the  high 
control  bandwidths  required  for  simulating  contact  with  virtual 
environments.  Unfortunately,  motors  have  very  low  power- 
to-weight  ratios,  which  tend  to  limit  the  force  output  of 
the  exoskeleton  for  physical  therapy  application.  Pneumatic- 
actuators,  on  the  other  hand,  have  high  power-to-weight  ratios 
but  poor  actuator  response,  rendering  them  too  bandwidth- 
limited  for  functional  rehabilitation. 

This  work  builds  upon  advances  in  actuator/drive  technol¬ 
ogy  to  develop  a  lightweight  but  powerful  exoskeleton  that  can 
be  used  for  exercise  therapy  and  functional  rehabilitation.  The 
article  begins  with  a  brief  survey  of  previous  arm  exoskeletons 
in  Section  II.  The  kinematic  and  mechanical  designs  are 
discussed  in  Section  III.  The  exoskeleton  control  system  and 
operational  modes  are  discussed  in  Section  IV.  The  software 
architecture  and  safety  system  are  addressed  in  Section  V. 
Some  conclusions  are  observed  in  Section  VI,  and  the  status 
of  the  exoskeleton  development  is  reviewed. 


II.  Previous  Work 

In  order  to  closely  follow  the  motion  of  the  human  arm, 
exoskeletons  are  typically  designed  with  the  seven  principal 
degrees  of  freedom  (DOF)  of  the  human  arm:  shoulder  (3), 
elbow  (1),  and  wrist  (3).  The  sequence  of  rotations  shown 
in  Figure  1  reflects  the  convention  commonly  used  in  the 
biomechanics  community  [18].  The  glenohumeral  (GH)  joint 
is  modeled  as  a  3 -DOF  ball  and  socket  joint  defined  by  a  set  of 
rotating  coordinates.  The  first  joint,  shoulder  flexion/extension, 
is  defined  as  the  rotation  of  the  shoulder  about  an  axis  through 
the  GH  joint  and  perpendicular  to  the  longitudinal  body  axis. 
The  second  axis,  shoulder  abduction/adduction,  is  the  rotation 
of  the  upper  arm  toward/away  from  the  body  about  an  axis  that 
is  perpendicular  to  the  flexion  axis  and  the  longitudinal  axis 
through  the  upper  arm.  The  third  axis,  shoulder  medial/lateral 
(intemal/extemal)  rotation,  is  the  roll  about  the  longitudinal 
axis  of  the  upper  arm.  Note  that  this  GH  model  is  purely 
rotational;  it  does  not  include  translation  of  the  glenohumeral 
joint  caused  by  scapulothoracic  motion,  clavicle  rotation,  and 
other  joints  in  the  shoulder  [13]. 


Fig.  1.  The  seven  principal  degrees  of  freedom  of  the  human  arm  (adapted 
from  [18]). 
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Alternatively,  several  bioengineering  researchers  have  fa¬ 
vored  the  “azimuth-elevation-roll”  convention  commonly  used 
in  scientific  practice  [23].  In  this  standard,  the  first  shoulder 
rotation,  “azimuth,”  is  the  rotation  about  a  vertical  axis  through 
the  shoulder.  The  second  rotation,  “elevation,”  is  rotation  of 
the  upper  arm  about  an  axis  orthogonal  to  both  the  azimuth 
axis  and  the  longitudinal  axis  of  the  upper  arm.  The  third 
shoulder  axis,  “roll,”  is  the  rotation  of  the  upper  arm  about  its 
longitudinal  axis  and  is  the  same  as  the  “shoulder”  rotation  in 
the  biomechanics  convention. 

Several  of  the  arm  exoskeleton  prototypes  that  have  been 
built  to-date  are  listed  in  Table  I.  The  table  lists  the  number 
of  joints,  power  source,  mass,  upper/forearm  lengths,  and  the 
shoulder  type.  If  the  exoskeleton  is  portable,  then  the  mass  of 
the  backpack  and  exoskeleton  are  each  given.  The  two  lengths 
reported  are  for  the  upper  arm  and  forearm  links.  Four  different 
shoulder  types  appear  based  upon  the  sequence  of  rotations 
in  the  shoulder.  The  range  of  motion  and  continuous  static 
torques  for  several  of  these  devices  are  listed  in  Table  II  and 
Table  III. 

The  majority  of  exoskeletons  listed  in  Table  I  were  devel¬ 
oped  as  haptic  devices  for  virtual  reality  (VR)  applications. 
Haptic  devices  are  typically  driven  by  motors  to  provide  the 
high  control  bandwidth  required  for  interaction  with  virtual 
environments  and  thus  have  relatively  low  power  output  as 
seen  in  Table  III.  The  only  exoskeleton  that  even  comes 
close  to  human  output  capability  is  the  hydraulically-powered 
Sarcos  Dextrous  Arm  Master  developed  as  a  force-reflecting 
master  arm  for  teleoperation  applications  [11].  While  this  may 
be  the  most  powerful  exoskeleton,  it  is  also  the  heaviest. 

TABLE  I 

Arm  exoskeleton  prototypes. 


DEVICE 

DOF 

# 

Port? 

Y/N 

Power 

* 

Masst 

(kg) 

Length* 

(cm) 

Type 

A 

EXOS  [4] 

5 

Y 

E 

8. 2/1. 8 

1/1 

FAR 

Dex  [11] 

7 

N 

H 

20.9 

31.1/25.9 

FAR 

Sensor  [21] 

7 

N 

E 

6 

24.0/28.5 

FAR 

GIA  [2],  [16] 

5 

N 

E 

10 

30.5/25.0 

AFR 

ATHD  [8] 

7 

Y 

E 

1/23 

1/1 

BSR 

MB  [22] 

7 

Y 

0 

1/ 15 

28.3/26.5 

ZLR 

FreFlex  [10] 

7 

N 

E 

? 

37.2/29.9 

AFR 

pMA  [25] 

7 

N 

P 

2 

1/1 

FAR 

Salford  [5] 

9 

Y 

0 

7/0.75 

1/1 

AFR 

MULOS  [12] 

5 

N 

E 

2 

1/1 

ZLR 

UWash  [24] 

7 

N 

E 

? 

1/1 

AFR 

*  E-electric,  H-hydraulic,  P-pneumatic,  0-unactuated 
1  Backpack/ Arm  (from  first  GH  joint) 

*  Upper  Arm  (GH  to  Elbow)/Forearm  (Elbow  to  Wrist) 

A  FAR  (flexion-abduction-rotation),  AFR  (abduction-flexion-rotation),  ZLR 
(azimuth-elevation-roll),  BSR  (ball&socket-rotation) 


A  number  of  unactuated  devices  have  also  been  built  for 
gathering  anthropomorphic  data  such  as  the  MB  Exoskeleton 
developed  for  the  U.S.  Air  Force  and  shown  in  Figure  2 
[22].  Although  this  device  is  passive,  it  incorporated  a  number 
of  features  important  for  physiotherapy  applications  such  as 
good  range  of  motion,  adjustable  link  lengths  (±2.5  cm  upper 


arm,  ±2.0  cm  forearm),  and  portability.  The  project  was 
discontinued  before  a  powered  exoskeleton  was  built,  but  first¬ 
hand  observation  provided  a  lot  of  valuable  information. 


Fig.  2.  SSF  personnel  dons  the  MB  Exoskeleton  during  visit  to  Wright- 
Patterson  Air  Force  Base.  (SSL  Photo  Archives  -  used  with  permission  of  the 
Wright-Patterson  Air  Force  Research  Laboratory) 

The  tradeoff  between  power-to-weight  ratio  and  control 
bandwidth  for  haptic  devices  has  been  addressed  by  several 
researchers  [4].  Recent  articles  suggest  a  trend  toward  using 
pneumatically  powered  exoskeletons  for  physical  therapy.  Ex¬ 
amples  of  these  include  the  pMA  Exoskeleton  which  utilizes 
pneumatic  muscle  actuators  (pMA)  [25]  and  the  Skil  Mate 
wearable  elbow/forearm  exoskeleton  powered  by  McKibben 
artificial  muscles  [26]  and  developed  for  astronaut  extravehic¬ 
ular  activity  (EVA).  While  these  devices  have  excellent  power- 
to-weight  ratios,  they  have  relatively  low  bandwidth  capability 
(«  0.5 Hz),  making  them  poorly  suited  at  present  for  virtual 
reality  applications.  However,  they  do  show  excellent  promise 
as  assistive  and  resistive  training  devices. 

The  Motorized  Upper  Limb  Orthotic  System  ( MULOS )  is  a 
wheelchair-mounted  exoskeleton  developed  for  use  by  persons 
with  weak  upper  limbs  [12];  thus,  it  is  not  intended  as  an 
exercise  system  for  fit  adults.  In  addition,  there  is  no  compen¬ 
sation  for  scapulo-thoracic  motion,  which  is  considered  key  for 
shoulder  rehabilitation.  Nonetheless,  MULOS  provided  some 
valuable  guidelines  for  designing  the  shoulder  kinematics  as 
well  as  instituting  a  number  of  novel  safety  features,  such  as 
a  slip  clutch  for  protection  against  spastic  motions. 

The  only  exoskeleton  that  has  explicitly  allowed  for  scapu- 
lothoracic  motion  is  the  non-driven  Salford  ArmMaster  de¬ 
veloped  for  tactile  VR  applications  [5],  [6].  This  exoskeleton 
incorporates  scapula  tilt  of  up  to  60°  and  scapula  medial 
rotation  of  up  to  45°  that  could  generate  up  to  12  cm  of 
scapula  elevation/depression.  MULOS  researchers  examined 
the  translation  of  the  GH  joint  for  several  assistive  tasks  and 
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deemed  that  the  motion  was  not  critical  in  their  application  [3]. 
The  Japanese-designed  ATHD  has  used  flex  cables  to  drive  the 
shoulder  rotational  degrees  of  freedom  from  motors  mounted 
in  a  backpack,  an  approach  that  could  allow  for  motion  of  the 
GH-joint,  but  only  at  the  expense  of  shoulder  rotation. 

TABLE  II 

Exoskeleton  Joint  Ranges  of  Motion  (deg). 


JOINT 

Man 

[18]+ 

Exos 

[4] 

Dex 

[4] 

Fre 

[10] 

GIA 

[2] 

Sen 

[20] 

HD 

[8] 

MB 

[22] 

shoulder 

flex/ext 

188 

/61 

120 

180 

130 

/52 

55 

/36 

150 

/30 

180 

/50 

130 

shoulder 

abd/add 

134 

/48 

120 

180 

28 

/is 

73 

/73 

50 

/o 

180 

/o 

135 

shoulder 

med/lat 

97 

/34 

100 

180 

90 

/90 

77 

/si 

60 

/60 

90 

/90 

260 

elbow 

flex/ext 

142 

/o 

100 

105 

166 

/-3 

89 

/15 

90 

/o 

115 

/o 

135 

forearm 

pro/sup 

85 

/90 

100 

105 

90 

/90 

99 

/88 

90 

/90 

90 

/90 

215 

wrist 

flex/ext 

90 

/99 

' 

180 

38 

/39 

50 

/20 

60 

/60 

70 

/90 

90 

wrist 

abd/add 

47 

/27 

100 

57 

/52 

80 

/80 

15 

/15 

55 

/25 

30 

^  Mean  values  for  dominant  arm  of  39  males 


TABLE  III 

Exoskeleton  Maximum  Joint  Torques  (N-m). 


JOINT 

Human 
[9]f  [1]* 

Exos 

[4] 

Dex 

[4] 

Fre 

[10] 

GIA 

[16] 

pMA 

[25] 

shoulder 

flex/ext 

115 

/no 

6.4 

97 

34 

20 

30 

shoulder 

abd/add 

134 

/94 

6.4 

97 

34 

20 

27 

shoulder 

med/lat 

2.3 

50 

17 

10 

6 

elbow 

flex/ext 

72.5 

/42.1 

1.6 

50 

17 

10 

6 

forearm 

sup/pro 

9.1 

/7.3 

0.4 

50 

5.6 

2 

5 

wrist 

flex/ext 

19.8 

/10.2 

5.5 

2.8 

4 

wrist 

abd/add 

20.8 

/17.8 

“ 

5.5 

2.8 

' 

4 

^Mean  values  for  male  shoulder,  dominant  arm 
-l- Mean  values  for  male  elbow/wrist,  dominant  arm 


III.  Mechanical  Design 

A  driving  goal  for  the  Maryland-Georgetown-Army  (MG A) 
Exoskeleton  was  to  use  the  lowest  number  of  DOFs  possible 
to  allow  for  full  exercise  therapy  of  the  shoulder  complex. 
Considering  the  design  from  a  biomechanical  perspective,  it 
was  determined  that  five  powered  degrees  of  freedom  would 
be  required.  Based  on  first  hand  experience  of  several  physical 
therapists,  allowing  for  incidental  or  deliberate  motion  of  the 
scapulothoracic  joint  would  be  critical  to  achieving  our  goals 
so  a  scapula  joint  was  included  in  the  design.  Three  joints 
would  be  required  for  glenohumural  shoulder  rotation.  An 
elbow  joint  was  needed  since  several  flexion/extension  muscles 
(triceps  and  biceps  brachii)  are  articulated  through  the  shoulder 


[13].  The  inclusion  of  the  elbow  implied  the  addition  of  a 
forearm  linkage,  so  the  forearm  roll  was  added  as  a  passive 
joint.  The  following  sections  will  review  the  development 
of  the  kinematics,  construction  of  the  rapid-prototypes,  and 
selection  of  the  hardware. 

A.  Kinematics 

Two  major  issues  arose  in  the  design  of  the  exoskeleton: 
how  to  articulate  translation  of  the  shoulder  joint  and  where 
to  place  the  shoulder  singularity.  It  was  clear  that  designing 
a  device  to  fully  articulate  11  or  more  DOFs  of  the  shoulder 
complex  would  not  be  reasonable  [19],  so  our  goal  was  to 
use  a  single  rotary  joint  as  a  first  order  approximation  of 
shoulder  translation.  In  addition,  the  singularity  or  “gimbal 
lock”  that  results  from  using  three  single-axis  shoulder  joints 
had  to  be  addressed.  The  intent  was  to  place  the  singularity  at 
an  azimuth  and  elevation  that  would  be  least  likely  to  interfere 
with  rehabilitation  tasks. 

Since  our  goal  was  to  capture  as  much  of  the  scapulotho¬ 
racic  motion  as  possible  using  a  single  rotary  joint,  we  decided 
to  focus  on  the  largest  motion,  which  is  shoulder  elevation  and 
depression.  Shoulder  elevation  can  occur  either  deliberately, 
e.g.  shoulder  shrug,  or  incidentally  during  rotation  of  the 
glenohumeral  (GH)  joint.  An  example  of  the  latter  occurs 
during  shoulder  abduction  starting  from  the  arm  hanging 
straight  down  by  the  side  of  the  leg  and  moving  in  a  circular 
arc  above  the  head  like  in  a  jumping  jack  motion. 

Figure  3  shows  the  translation  of  the  GH  joint  in  the  frontal 
plane  and  was  generated  by  using  video  capture  data  from  a 
shoulder  abduction  movement  [19].  The  vertical  displacement 
(elevation)  of  the  shoulder  was  found  to  be  far  greater  than 
the  horizontal  (protraction)  displacement,  with  a  sharp  point 
in  the  curve  where  the  adduction  angle  reaches  90°  (arm  is 
horizontal).  This  point  corresponds  to  the  onset  of  rotation  of 
the  clavicle,  which  then  thrusts  the  acromio-clavicular  (AC) 
joint  and  consequently  the  GH  joint  as  the  shoulder  continues 
to  elevate. 


Elevation  (mm) 

Fig.  3.  Displacement  of  shoulder  in  frontal  plane  during  a  180°  shoulder 
abduction  using  data  taken  from  [19]  (curve  fit  is  2nd  order  polynomial). 

If  a  circular  curve  could  be  fit  to  data  similar  to  Figure  3,  an 
axis  of  rotation  could  be  identified  for  placement  of  a  scapu¬ 
lothoracic  joint.  Since  the  axis  of  this  joint  is  perpendicular  to 
the  frontal  plane,  it  could  be  mounted  anywhere  along  a  line 
normal  to  the  back.  Therefore,  the  motor  can  be  conveniently 
mounted  on  a  backpack  strapped  to  the  subject’s  torso  and 
can  yield  the  desired  elevation  and  depression  of  the  shoulder 
joint. 
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Shoulder  singularities  were  the  second  major  issue  to  be 
confronted.  The  GH  joint  is  usually  modeled  kinematically 
as  a  ball  and  socket  joint.  However,  attempting  to  implement 
the  same  type  of  joint  in  the  exoskeleton  would  pose  severe 
design  challenges  as  well  as  introduce  interference  between  the 
human  joint  and  robotic  joint.  Instead,  the  ball  and  socket  joint 
is  replaced  by  three  serially  connected  pin  joints.  Although 
the  pin  joints  create  three  DOFs  around  a  single  point,  they 
do  not  exactly  replicate  the  kinematics  of  a  ball  and  socket 
joint.  Most  importantly,  the  series  of  pin  joints  create  two 
singularities,  180°  apart  from  one  another.  These  singularities 
can  be  moved,  but  cannot  be  eliminated. 

Design  of  the  elbow  joint  is  considerably  more  straightfor¬ 
ward  because  it  can  be  approximated  as  a  single  pin  joint.  The 
actuator  corresponding  to  this  joint  will  incorporate  a  torque- 
limiting  slip  clutch,  which  decouples  the  actuator  from  the 
frame  of  the  robotic  arm  if  a  predetermined  torque  value  is 
exceeded.  This  device  will  help  protect  the  user  from  injury 
by  allowing  free  movement  in  the  elbow  if  a  spasm  occurs  in 
that  joint. 

The  forearm  roll  will  be  the  terminal  joint  on  this  prototype. 
It  will  be  equipped  with  an  encoder  to  measure  the  joint  angle, 
but  will  not  be  driven.  A  mounting  bracket  for  a  hand  grip 
will  be  rigidly  attached  to  the  forearm  link  through  a  six-axis 
force/torque  sensor.  Thus  the  wrist  abduction  and  flexion  joints 
will  be  fixed  in  the  first  hardware  prototype. 

B.  Rapid-Prototyped  Models 

Two  rapid-prototyped  (RPT)  models  of  the  MGA  Exoskele¬ 
ton  were  built.  The  first  prototype  was  used  mainly  to  evaluate 
the  kinematics  and  to  decide  what  passive  link  adjustments 
would  be  required.  The  second  prototype  reflected  kinematics 
changes  following  a  design  review  and  closely  matched  the 
final  design.  The  prototypes  were  attached  to  the  back  of  a 
neopreme  Uni-Vest  ™  weight  vest  for  portability,  and  a  set 
of  passive  adjustments  between  the  scapula  and  shoulder  joints 
allowed  for  a  customized  fit. 

1 )  Prototype  I:  A  3D  model  of  the  first  prototype  is  shown 
in  Figure  4.  An  adjustable  linkage  connects  the  scapulo- 
thoracic  joint  to  the  first  shoulder  pin  joint  which  is  oriented 
horizontally  as  seen  in  Figure  5.  This  segment  has  two 
angular  adjustments  accounting  for  curvature  in  the  back,  and  a 
prismatic  adjustment  accounting  for  varying  distance  between 
the  scalupo-thoracic  joint  and  the  GH  joint. 

The  second  joint  axis  was  mounted  orthogonally  to  the 
first,  and  the  third  joint  axis  was  mounted  orthogonally  to 
the  second.  When  the  three  GH  joint  axes  become  coplanar, 
the  shoulder  becomes  singular.  For  an  orthogonal  triad,  this 
singularity  occurs  when  the  first  and  third  joint  axes  align 
producing  a  singularity  along  the  direction  of  the  first  joint 
axis.  The  direction  of  this  singularity  can  be  changed  using 
the  second  angular  adjustment  in  the  first  linkage.  Since 
this  passive  adjustment  is  oriented  vertically,  the  singularity 
remains  in  the  transverse  plane. 

The  third  GH  joint  axis  is  the  shoulder  intemal/external 
rotation.  A  “C”-shaped  linkage  fits  over  the  upper  arm  as 


shown  previously  in  Figure  5.  A  strap  secures  a  rotational  joint 
on  this  linkage  to  the  arm  so  that  the  joint  moves  with  the  arm. 
Modifications  to  the  design  of  the  first  prototype  began  before 
the  elbow  linkage  was  constructed. 


A 

f.  Z.^—^ 

V  A 

4ft  \ 

jFo  4 

•r _ 

Fig.  4.  CAD  schematic  of  Prototype  I. 


Fig.  5.  Side  view  of  Prototype  I  with  elbow  and  shoulder  at  90°  flexion. 

2)  Prototype  II:  A  schematic  of  the  second  prototype  is 
shown  in  Figure  6.  Significant  kinematic  changes  were  imple¬ 
mented  from  the  first  prototype  to  move  the  singularity  out  of 
the  transverse  plane  into  a  less  intrusive  location.  As  in  the  first 
prototype,  an  adjustable  linkage  connects  the  scapulothoracic 
joint  to  the  first  GH  joint  axis  as  seen  in  Figure  7.  However,  an 
additional  passive  rotation  axis  in  the  transverse  plane  is  used 
to  rotate  the  first  shoulder  axis  (and  thus  singularity)  about 
30°  from  a  normal  to  the  plane  containing  the  scapula  axis 
and  the  GH  joint. 

Keeping  the  first  axis  away  from  the  normal  allows  the 
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C.  Hardware 

The  objective  of  the  mechanical  design  was  to  achieve  at 
least  half  human  strength  and  90%  of  the  human  range  of 
motion  in  each  joint  given  previously  in  Tables  II  and  III. 
The  realized  stall  torque  and  range  of  motion  for  the  active 
joints  are  given  in  Table  IV  and  meet  these  specifications.  A 
drawing  of  the  final  exoskeleton  design  is  shown  in  Figure  8. 

TABLE  IV 

Joint  characteristics  for  MG  A  Exoskeleton. 


Fig.  8.  IDEAS-CAD  rendering  of  final  exoskeleton  design. 

Power  and  communication  is  routed  through  an  umbilical 
using  a  Galil  6-Axis  Motion  Control  card  mounted  in  the  PC 
and  Advanced  Motion  Control  PWM  power  modules.  Motor 
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arm  to  rest  straight  down  against  the  side  without  being 
in  a  singular  configuration.  This  alignment  of  the  shoulder 
singularity  with  the  azimuth  axis  in  the  MB  Exoskeleton 
was  very  problematic  during  our  examination  (see  Figure  2). 
Realignment  of  the  azimuth  axis  was  also  employed  in  the 
MULOS  design,  although  the  tilt  was  30°  from  vertical  in  the 
sagittal  rather  than  frontal  plane. 

The  second  GH  joint  axis  is  oriented  90°  from  the  first  and 
is  located  to  the  side  and  slightly  beneath  the  shoulder,  as 
shown  in  Figure  7.  The  third  GH  joint  is  perpendicular  to  the 
second,  but  makes  a  135° -angle  with  the  first  when  the  arm  is 
in  the  rest  position.  The  reason  for  making  this  angle  135°  and 
not  90°  is  to  allow  the  arm  to  abduct  further  before  the  first 
and  third  GH  joints  interfere  with  each  other.  Although  the 
elbow  linkage  was  not  built  for  the  prototype,  the  attachment 
to  the  upper  arm  can  be  seen  at  the  base  of  Figure  7. 


Fig.  6.  CAD  schematic  of  Prototype  II. 


Fig.  7.  Prototype  II  is  shown  at  90°  shoulder  flexion  while  mounted  to  a 
weight  vest. 


Each  joint  (except  for  the  forearm)  is  driven  by  a  Koll- 
morgen  brushless  DC  motor  and  harmonic  drive  transmission 
from  HD  Systems.  Harmonic  drives  were  chosen  because 
of  their  compactness,  low  backlash,  and  backdrivability.  The 
elbow  is  equipped  with  a  slip  clutch  manufactured  by  Mayr 
Power  Transmission  and  has  an  adjustable  torque  range  of 
20  —  50  N-m.  It  also  features  a  synchronous  mechanism,  which 
restricts  the  device  from  recoupling  in  all  but  one  position,  thus 
preserving  the  orientation  between  the  input  and  the  output. 


= 

30° 

Ve  = 

45° 

kg  “ 

19.5-26.8  cm 

27.2-31.2  cm 

kflw  — 

23-32  cm 

Joint 

Gear 

Ratio 

Stall  Torque 
(N-m) 

Range 

(deg) 

Scapula 

160  :  1 

92 

+30/  -  30 

GHR  1 

160  :  1 

92 

+180/  -  45 

GHR  2 

160  :  1 

92 

+90/  -  65 

GHR  3 

160  :  1 

92 

+210/  -  30 

Elbow 

160  :  1 

64 

+142/0 
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position  is  determined  using  an  1800-line  optical  incremental 
encoder  manufactured  by  Numerik  Jena.  Single-turn,  12-bit 
optical  absolute  encoders  manufactured  by  Gurley  Precision 
Instruments  are  mounted  at  the  output  of  the  transmission  to 
determine  absolution  position  on  start  up  and  to  monitor  the 
incremental  encoders. 

A  Model  50M31A  force/torque  sensor  manufactured  by 
JR3,  Inc.  is  used  to  measure  forces  and  torques  on  the  handle. 
The  sensor  has  a  rating  of  25  lb  (111  N)  in  the  radial  direction 
and  50  lb  (222  N)  along  the  axial  direction.  The  unit  is  a  very 
compact  50  mm  x  31  mm  and  has  a  mass  of  0.15  kg.  The 
sensor  has  integrated  sensor  electronics,  and  the  six-channel 
digital  output  is  read  by  a  PCI  card  at  8  KHz.  A  single-axis 
torque  sensor  manufactured  by  Omega  Engineering,  Inc.  is 
attached  to  the  output  side  of  the  scapula  transmission.  The 
sensor  is  capable  of  measuring  torque  levels  up  to  113  N-m. 
Two  single-axis  compression  load  cells  made  by  Sensotec,  Inc. 
are  attached  to  mounting  plates  on  either  side  of  the  elbow  to 
measure  axial  load  at  the  elbow.  Each  sensor  is  capable  of 
measuring  forces  up  to  25  lb  (111  N). 

IV.  Control  System 

The  exoskeleton  operates  in  two  modes:  Virtual  Reality 
(VR)  Mode  and  Physical  Therapy  (PT)  Mode.  In  VR  Mode, 
the  forces  exerted  at  the  hand  are  controlled  by  interaction  with 
a  virtual  environment  generated  by  a  computer.  In  PT  Mode, 
the  arm  is  allowed  to  rotate  about  an  arbitrary  axis  through 
the  shoulder  using  a  preset  resistance  profile.  In  either  mode, 
the  scapula  joint  moves  independently  to  “accommodate”  the 
patient  using  sensed  torque  from  the  local  torque  cell.  The 
control  of  the  arm  joints  and  scapula  joint  are  described  in 
more  detail  below. 

A.  Virtual  Reality  Mode 

In  Virtual  Reality  (VR)  Mode,  computer-generated  environ¬ 
ments  are  used  to  simulate  daily  living  tasks  for  functionally- 
based  rehabilitation.  The  patient  views  the  simulated  task 
and  representation  of  their  arm  through  a  head  mounted 
display  while  the  exoskeleton  provides  haptic  feedback  to  the 
patient.  A  force  sensor  located  at  the  hand  gripper  senses  the 
forces  being  exerted  by  the  patient’s  “contact”  with  the  virtual 
environment  and  relays  them  to  the  controller  which  moves 
the  exoskeleton  in  response  to  the  interaction. 

Because  the  exoskeleton  is  kinematically  redundant,  speci¬ 
fication  of  the  wrist  position  is  not  enough  to  specify  the  three 
shoulder  rotations  and  elbow  pitch.  This  self-motion  manifests 
itself  as  the  ability  of  the  elbow  to  “orbit”  about  the  shoulder- 
wrist  axis  pw  shown  in  Figure  9  while  the  position  of  the 
wrist  and  shoulder  are  held  fixed  [14].  The  angle  formed  by 
the  shoulder-elbow-wrist  (SEW)  plane  with  a  reference  vector 
v  is  referred  to  as  the  “SEW  angle.” 

The  admittance  controller  shown  in  Figure  10  is  used  to 
convert  the  sensed  contact  forces  at  the  hand  and  elbow  into 
desired  movements  of  the  exoskeleton  [7],  [15].  Signals  from 
the  force-torque  sensor  at  the  hand  are  relayed  to  an  admittance 
model  of  the  virtual  environment,  which  then  outputs  a  desired 


Fig.  9.  Definition  of  shoulder-elbow-wrist  (SEW)  roll  angle  and  location  of 
gripper  force/torque  sensor  and  elbow  axial  load  cells. 

velocity  for  the  wrist,  pw.  In  addition,  a  pair  of  compression 
load  cells  mounted  along  the  elbow  axis  are  used  to  determine 
the  torque,  r^,  exerted  about  the  shoulder- wrist  axis,  pw.  The 
SEW  torque  is  then  integrated  to  produce  a  desired  SEW  roll 
velocity  <j)d  that  is  proportional  to  the  torque  about  the  pw 
axis.  The  desired  wrist  and  SEW  velocities  are  then  converted 
to  desired  angular  velocities  at  the  joints  using  the  inverse 
Jacobian,  JW($>,  and  the  resultant  desired  joint  angles,  0d,  are 
then  tracked  using  a  proportional-derivative  (PD)  control  law. 


join!  pjuouifmieR 


Fig.  10.  Admittance  controller  used  in  VR  Mode. 

B.  Physical  Therapy  Mode 

In  Physical  Therapy  (PT)  Mode,  the  exoskeleton  becomes 
a  programmable  resistance  trainer  that  allows  the  patient  to 
exercise  about  an  arbitrary  shoulder  rotation  axis.  For  rotator 
cuff  injuries,  for  example,  therapists  often  prescribe  exercises 
involving  lateral/medial  rotation  of  the  shoulder.  Since  there  is 
no  single  joint  corresponding  to  shoulder  rotation,  the  exercise 
involves  all  three  shoulder  axes  of  the  exoskeleton.  Thus,  the 
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controller  needs  to  yield  a  prescribed  resistance  profile  about 
the  desired  axis  while  preventing  rotations  about  the  other 
shoulder  axes. 

Since  the  torques  about  the  shoulder  axes  cannot  be  mea¬ 
sured  directly,  an  impedance  controller  is  used  to  realize  the 
desired  resistance  profile  [17].  The  shoulder  joint  velocities 
are  relayed  to  a  Jacobian,  Jqh ,  as  shown  in  Figure  11,  which 
then  computes  the  Cartesian  velocities  about  the  glenohumeral 
(GH)  joint,  ooQHd'  These  velocities  are  then  multiplied  by  the 
desired  resistance  profile,  which  outputs  the  desired  torques 
about  the  GH  axes,  Tend-  These  torques  are  then  converted 
into  exoskeleton  joint  torques  using  the  Jacobian.  A  feedfor¬ 
ward  model  of  the  exoskeleton  runs  in  parallel  to  calculate 
gravity  and  other  feedforward  compensation  torques.  Since 
only  the  GH  angles  are  specified,  the  elbow  pitch  is  left 
unconstrained  and  can  be  moved  however  the  patient  desires. 


Fig.  11.  Impedance  Controller  used  in  PT  Mode. 


C.  Scapula  Joint  Control 

The  scapula  joint  is  controlled  independently  from  the  arm 
joints  using  the  admittance  controller  shown  in  Figure  12.  A 
torque  cell  at  the  output  of  the  transmission  directly  measures 
the  torque  being  exerted  by  the  scapula  joint,  T£.  However, 
because  the  applied  torque  must  balance  the  sum  of  the 
gravitational  torque  and  the  torque  applied  by  the  subject, 
a  gravitational  model  is  used  to  subtract  out  the  component 
due  to  the  gravity  load.  The  estimated  human  torque  is  then 
integrated  to  produce  a  desired  scapula  joint  velocity,  £<*, 
which  is  then  tracked  by  a  PD  controller. 


Fig.  12.  Accommodation  controller  used  for  scapula  joint. 


V.  Software  Architecture 

The  architecture  of  the  control  system  is  shown  in  Figure  13. 
The  health  worker  interacts  with  the  system  through  the 


control  station  and  its  user  interface.  This  interface  allows 
the  clinician  to  monitor  the  robotic  system  performance,  enter 
patient  rehabilitation  parameters,  and  store  patient  history. 
The  control  station  communicates  over  the  Internet  with  the 
robot  control  computer,  which  is  responsible  for  control  of  the 
robotic  arm  and  overall  patient  safety.  The  arm  controllers  run¬ 
ning  on  the  robot  control  computer  produce  either  a  position 
or  torque  command,  which  is  sent  to  the  motor  controller.  A 
force-torque  sensor  attached  to  the  gripper  senses  the  forces 
and  torques  exerted  by  the  patient,  which  are  relayed  to  the 
robot  control  computer  via  a  digital  acquisition  board. 

The  robot  control  computer  runs  the  TimeSys  Linux  real¬ 
time  operating  system,  in  order  to  guarantee  meeting  its  safety 
deadlines.  As  the  operator  computer  is  not  involved  in  safety 
decisions,  it  will  run  a  standard  desktop  operating  system 
(in  which  the  ability  to  respond  to  events  within  a  certain 
time  frame  is  not  guaranteed).  The  arm  controller  algorithms 
operate  in  Cartesian  space,  utilize  force/torque  sensor  data, 
and  operate  at  200  Hz.  The  PD  controller,  when  required  by 
a  particular  arm  controller,  runs  at  1000  Hz. 


Control  station 


Fig.  13.  Software  control  architecture. 

All  safety  decisions  are  carried  out  within  the  robot  control 
computer  and  occur  autonomously.  This  setup  allows  for  very 
high  speed  reaction  by  the  computer  safety  system  in  the  event 
of  a  component  failure,  communication  error,  or  the  patient 
attempting  something  they  should  not.  The  embedded  robot 
system  is  designed  to  be  a  ‘fail-safe’  system,  and  is,  as  much 
as  practical,  safe  in  and  of  itself.  The  actual  ‘safe  state’  entered 
will  be  one  of:  a)  the  arm  holding  its  current  position  and  not 
exerting  any  force  on  the  patient,  or  b)  a  complete  power-down 
of  the  arm.  At  any  time,  either  the  clinician  or  the  patient  are 
able  to  manually  safe  the  system  by  hitting  a  button  or  key. 
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The  robot  control  computer  is  responsible  for  enforcing 
patient-related  force,  position,  and  range-of-motion  limits; 
guaranteeing  that  the  heartbeat  transmitted  by  every  computer 
is  valid;  verifying  that  all  the  local  electronics  are  functioning 
and  correctly  providing  data;  and  ensuring  that  the  operator 
computer  is  functioning.  These  checks  occur  at  either  200 
or  1000  Hz,  depending  on  whether  they  are  associated  with 
the  arm  or  PD  controller.  Also,  the  robot  control  computer  is 
protected  by  watchdog  timers  that  will  safe  the  system  in  the 
event  of  incorrect  operation. 

The  reaction  time  of  the  software  safety  system  determines 
how  much  affect  a  failure  can  have  on  the  patient,  in  terms  of 
how  much  additional  force  can  be  applied  or  how  far  the  arm 
can  move,  before  the  system  safes.  This  effect  depends  on  the 
reaction  time  of  the  robot  control  computer  and  the  rate  of  the 
particular  safety  check.  As  software  checks  are  executed  at 
either  200  or  1000  Hz,  the  reaction  time  will  be  a  maximum 
of  5  ms,  and  a  minimum  of  1  ms. 

VI.  Conclusion 

This  paper  began  with  a  survey  of  exoskeletons  built  to- 
date  and  has  explored  the  design  tradeoffs  between  various 
kinematic  designs  and  actuator  technologies.  Only  one  ex¬ 
oskeleton  was  found  that  incorporated  scapulothoracic  motion, 
but  it  was  not  powered.  A  single  rotary  joint  perpendicular  to 
the  back  was  chosen  to  accommodate  shoulder  elevation  and 
depression.  The  glenohumeral  joint  is  based  on  an  orthogonal 
axis  triad  with  the  first  axis  tilted  at  30°  away  from  the  azimuth 
axis  to  accommodate  the  singularity.  Two  rapid-prototyped 
versions  of  the  exoskeleton  were  built  prior  to  final  design. 
Mechanical  fabrication  is  complete,  and  the  exoskeleton  is 
now  undergoing  electronics  integration  and  testing.  Although 
the  prototype  mass  is  only  about  12  kg,  it  will  initially  be 
wall-mounted  due  to  the  weight  of  the  external  components. 
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ABSTRACT 

Open  source  software  has  tremendous  potential  for  improving  the  productivity  of  research  labs  and  enabling  the 
development  of  new  medical  applications.  The  Image-Guided  Surgery  Toolkit  (IGSTK)  is  an  open  source  software 
toolkit  based  on  ITK,  VTK,  and  FLTK,  and  uses  the  cross-platform  tools  CMAKE  and  DART  to  support  common 
operating  systems  such  as  Linux,  Windows,  and  MacOS.  IGSTK  integrates  the  basic  components  needed  in  surgical 
guidance  applications  and  provides  a  common  platform  for  fast  prototyping  and  development  of  robust  image-guided 
applications.  This  paper  gives  an  overview  of  the  IGSTK  framework  and  current  status  of  development  followed  by  an 
example  needle  biopsy  application  to  demonstrate  how  to  develop  an  image-guided  application  using  this  toolkit. 

Keywords:  open  source,  image-guided  surgery,  surgical  guidance,  tracking,  state  machine,  needle  biopsy,  application 
prototyping 


1.  PURPOSE 

The  field  of  image-guided  surgery  is  rapidly  expanding,  as  new  techniques  are  being  developed  that  minimize  treatment 
invasiveness  and  thereby  reduce  trauma  and  speed  recovery  for  patients.  A  typical  image-guided  system  consists  of 
tracking  subsystems  for  capturing  the  movement  of  instruments  and  the  patient,  and  a  computer  subsystem  for 
integrating  and  displaying  pre-  and  intra-operative  images  and  instrument  positions  using  the  tracking  data.  Most  of  the 
development  effort  is  focused  on  implementing  a  general  software  infrastructure  that  supports  the  basic  capabilities  and 
inter-operation  of  these  components.  As  new  trackers  and  data  integration  algorithms  are  developed,  maintaining  a 
general  infrastructure  that  nevertheless  allows  new  features  to  be  exploited  is  even  more  challenging,  as  most  software 
platforms  are  not  designed  to  be  extendable  from  the  outset.  Testing  such  systems  can  also  be  an  extensive  process,  and 
the  lack  of  proper  testing  expose  risks  to  patients  and  surgeons.  The  purpose  of  IGSTK  is  to  enable  the  creation  and 
validation  of  reusable,  robust  image-guided  software  as  open-source,  free  of  commercial  licensing  restrictions.  This 
component-based  software  toolkit  should  make  it  easy  for  researchers  to  prototype  and  develop  image-guided  surgery 
applications. 


2.  METHODS 

IGSTK  has  been  under  development  for  over  a  year  and  a  public  beta  release  will  be  available  by  the  time  of  this 
conference  (February  2006).  IGSTK  is  based  on  the  following  existing  open  source  software  toolkits:  ITK  for 
segmentation  and  registration  [1],  VTK  for  visualization  [2],  CMake  for  cross-platform  building  and  DART  for  cross- 
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platform  testing,  and  FLTK  for  the  user  interface  [3]  (though  the  user  interface  layer  is  designed  to  allow  for  future 
application  development  with  other  UI  toolkits).  The  IGSTK  toolkit  contains  the  basic  software  components  needed  to 
construct  an  image-guided  system,  including  support  for  trackers,  data  integration  algorithms,  and  a  four-quadrant  view 
incorporating  image  overlay. 

1 .  Given  that  IGSTK  is  intended  for  developing  applications  that  will  be  used  to  treat  patients,  robustness  and 
quality  are  the  highest  priorities  in  the  design  of  the  toolkit.  To  minimize  the  risk  of  harm  to  the  patient 
resulting  from  misuse  of  the  classes,  IGSTK  is  designed  based  on  the  following  principles  [4] : 

2.  Requirements  are  generated  by  studying  and  documenting  scenarios  of  the  surgical  procedures  in  a  clinical 
environment.  By  analyzing  these  scenarios,  features  that  are  necessary  for  fulfilling  the  requirements  are 
extracted  and  implemented. 

3.  All  IGSTK  components  are  governed  by  a  state  machine.  A  state  machine  is  contained  within  the  class  to 
control  the  access  to  the  class.  Components  are  always  in  a  valid  state  to  ensure  they  will  perform  in  a 
predictable  manner.  The  use  of  a  state  machine  also  helps  enforce  high  quality  standards  for  code  coverage  and 
run-time  validation. 

4.  IGSTK  does  not  directly  expose  ITK  and  VTK  objects  APIs  to  the  application  developer.  Instead,  the  ITK  and 
VTK  objects  are  tightly  encapsulated  by  IGSTK  objects  that  provide  a  minimal  but  sufficient  API  for 
application  development.  This  ensures  that  any  APIs  used  by  developers  have  been  specifically  tested  for 
image  guidance  applications. 

5.  IGSTK  does  not  utilize  dynamic  object  typing.  When  IGSTK  objects  are  created  they  must  be  declared  with 
full  specialization,  and  all  type  checking  is  performed  at  compile  time  and  never  at  run  time.  This  reduces  the 
number  of  software  parameters  that  can  be  adjusted  at  run-time,  and  hence  reduces  the  possibility  that  the 
software  will  be  running  in  an  untested  configuration. 

6.  IGSTK  uses  events  to  communicated  between  components,  where  the  events  are  produced  by  the  state 
machine  of  one  object  and  received  as  inputs  to  the  state  machine  of  one  (or  more)  other  objects.  The  mapping 
of  events  to  state  machine  inputs  is  explicitly  defined  and  deterministic. 

7.  IGSTK  does  not  use  a  return  value  or  exception  at  the  application  level.  Instead  it  uses  events  to  pass 
information  around. 

2.1  State  machine 

To  ensure  safety  and  robustness,  the  state  machine  design  pattern  is  incorporated  into  the  IGSTK  components.  A  state 
machine  is  defined  by  a  set  of  states,  a  set  of  inputs,  and  a  set  of  directed  transitions  between  states.  Transitions  are 
changes  from  one  state  to  another  when  a  certain  stimulus  or  input  is  received.  An  action  may  be  taken  along  with  a 
transition,  to  perform  some  task  on  entry  to  a  state,  or  on  exit  from  a  state.  State  machines  support  the  component-based 
architecture  through  encapsulation  and  extensibility.  They  ensure  that  a  component  is  always  in  a  valid  state  which  can 
guarantee  repeatable  and  deterministic  behavior.  A  description  of  the  state  machine  theory  can  be  found  in  [6]. 

The  following  points  summarize  the  advantage  of  using  state  machine  in  the  toolkit  [7]: 

1.  Safety  and  Reliability :  A  state  machine  ensures  that  component  behavior  is  deterministic  and  that  all 
components  are  in  a  known  and  error-free  state  at  any  given  moment. 

2.  Cleaner  design :  Since  developers  must  anticipate  all  possible  inputs,  states,  and  transitions,  the  state  machine 
encourages  and  enforces  a  cleaner  and  more  robust  design,  free  of  untested  assumptions. 

3.  API  simplicity:  A  focused,  clearly  expressed  application  programming  interface  (API)  is  a  must  for  supporting 
robustness  and  reliability.  In  the  context  of  surgical  guidance,  we  believe  that  flexibility  and  abundance  of 
features  are  undesirable,  because  they  create  more  opportunities  for  things  to  go  wrong  during  a  surgical 
intervention. 

4.  A  consistent  integration  pattern :  The  toolkit’s  value  as  it  matures  will  undoubtedly  be  tied  to  the  incorporation 
of  additional  functionality  at  the  component  level.  This  functionality  will  often  take  the  form  of  reusable  code 
from  existing  toolkits.  State  machines  provide  a  consistent  pattern  for  integrating  this  functionality  while 
adhering  to  the  safety- first  principles  necessitated  by  the  application  domain. 

5.  Quality  Control :  State  machines  facilitate  code  coverage  in  the  sense  of  lines  of  code  tested,  as  well  as  path 
coverage  on  a  per  component  basis.  Using  code  that  is  not  based  on  state  machines  may  result  in  applications 
that  exhibit  unreliable  behavior.  At  run  time,  they  can  easily  enter  into  any  number  of  untold  states  that  were 
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never  explored  by  the  developer,  and  can  lead  to  error  conditions  that  may  or  may  not  become  visible  to  the 
users  of  the  application. 

The  safety  emphasis  in  IGSTK  encourages  explicit  knowledge  of  component  state  and  determines  whether  a  given 
behavior  may  be  executed  while  in  that  state.  Hence,  our  application  is  a  combination  of  traditional  concepts  in 
reliability  (being  in  a  known  state)  and  the  State  Pattern  [8]  (for  managing  availability  of  behavior  dynamically). 

2.2  Architecture  and  components 

Figure  1  presents  a  UML  collaboration  diagram  of  the  major  IGSTK  components  involved  in  a  typical  image-guided 
surgery  application.  The  key  components  are  from  left  to  right  [4,  5]: 

1.  View  classes  are  used  for  presenting  results  and  displays  to  the  physician.  An  internal  pulse  generator  is  used 
to  update  the  scene  at  specified  refresh  rate  to  keep  the  visual  information  up-to-date.  Currently  the  IGSTK 
View  classes  are  built  with  VTK  classes  and  an  FLTK  window.  They  take  FL  events  and  translate  them  into 
VTK  events  to  enable  user  interaction  within  the  render  window.  View2D  and  View3D  are  subclasses  of  View 
with  a  different  interaction  style. 

2.  SpatialObject  and  SpatialObjectRepresentation.  SpatialObjects  classes  model  physical  objects  including  image 
data  and  simple  geometrical  shapes  such  as  anatomical  structures  and  surgical  devices,  while 
SpatialObjectRepresentations  classes  are  responsible  for  displaying  their  associated  SpatialObject  in  the  scene. 
IGSTK  Spatial  Objects  can  be  attached  to  a  Tracker  Tool  object,  so  that  it  can  update  its  position  as  the 
Tracker  Tool  moves.  The  state  machine  in  SpatialObjects  classes  enforces  this  association  to  be  a  one  time 
operation.  Once  a  Spatial  Object  is  attached  to  a  tracker  tool  it  is  not  expected  to  get  back  to  manual  control 
nor  to  be  re-attached  to  a  second  tracker  tool.  It  is  no  longer  possible  to  change  the  position  of  the  object 
programmatically.  IGSTK  has  a  variety  of  SpatialObjects  and  corresponding  SpatialObjectsRepresentations, 
including  basic  shapes  (Cone,  Cylinder,  Ellipsoid  and  so  on),  ImageSpatialObject,  UltrasoundProbeObject,  and 
GroupObject  to  make  composite  SpatialObjects. 

3.  Tracker  classes  are  an  abstraction  of  tracking  devices.  They  communicate  with  hardware  and  store  data 
including  position,  orientation,  and  other  relevant  information  pertaining  to  tracked  surgical  tools.  The  IGTSK 
Tracker  component  is  developed  based  on  code  donated  by  Atamai  Inc.  and  wrapped  with  a  state  machine.  The 
internals  of  the  Tracker  require  a  separate  thread  to  communicate  with  tracking  hardware  at  an  acceptable 
frequency.  A  pulse  generator  inside  the  class  is  used  to  automatically  update  the  location  information  of  the 
tracked  device  with  an  expiration  time  and  propagate  changes  to  SpatialObjects.  IGSTK  now  supports  NDI 
AURORA,  POLARIS,  and  VICRA  trackers  (Northern  Digital  Inc.,  Waterloo,  Canada)  and  Flock  of  Birds 
tracker  (Ascension  Technology  Corporation,  Burlington,  VT  USA),  although  not  all  features  of  these  trackers 
are  supported. 

IGSTK  Application  Viewers  SpatialObject  SpatialObjects  Tracker 

Representations 


Figure  1 :  IGSTK  Component  Architecture 
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Some  other  components  do  not  present  in  Figure  1  include:  DICOMImageReader  for  reading  in  DICOM  images, 
Landmark3 DRegistration  for  3D  pair-point-based  rigid  body  registration,  Logger  for  logging  debug  or  other 
information,  and  Calibration  class  for  calibrating  tracker  tools.  All  these  together  provide  essential  functionalities  to 
build  an  image-guided  system. 


3.  EXAMPLE  APPLICATION 

This  section  describes  the  procedure  to  develop  an  image-guided  biopsy  application  using  IGSTK.  A  typical  image- 
guided  needle  biopsy  procedure  involves  first  acquiring  a  pre-operative  CT  image  and  then  registering  the  CT  image  to 
the  patient  coordinate  system.  For  this  purpose,  fiducial-based  rigid  body  registration  techniques  are  commonly  used. 
During  the  biopsy  phase,  the  needle  is  tracked  by  an  optical  tracking  device  with  real-time  visualization  of  its  location 
overlaid  on  top  of  the  CT  image.  This  overlay  image  provides  guidance  to  the  surgeon  for  better  targeting  of  the  needle 
to  its  desired  location.  In  this  example,  state  machine  is  being  implemented  at  the  application  level.  While  this  is  not 
mandatory,  it  is  strongly  recommended  when  using  IGSTK.  A  state  machine  architecture  gives  the  application 
developer  an  easier  way  to  prototype  the  application  and  to  control  the  work  flow  of  the  surgical  procedure,  and  also 
adds  an  extra  layer  of  security  to  the  application  to  make  it  more  robust.  The  following  sections  demonstrate  how  to 
write  an  application  using  the  IGSTK  framework. 

3.1  Prototyping 

The  first  step  is  to  analyze  the  surgical  application  and  develop  a  least  a  minimal  specification.  By  analyzing  a  typical 
needle  biopsy  procedure,  we  identify  the  following  essential  tasks  in  this  procedure  [9]. 

1 .  Obtain  the  patient  demographic  information  (name,  etc.). 

2.  Load  in  the  pre-operative  CT  image  using  the  DICOM  file  format.  Fiducials  (small  markers)  are  usually  placed 
on  the  anatomy  prior  to  the  scan  for  landmark  based  registration  in  Steps  4-7. 

3.  Verify  the  patient  information  against  the  information  in  the  image.  Prompt  the  surgeon  if  there  is  a 
discrepancy.  This  step  is  typical  of  the  error  checking  that  should  be  done  and  one  should  assume  that  if 
anything  can  go  wrong  it  will  go  wrong  and  safeguards  should  be  provided. 

4.  Identify  the  image  landmarks  by  going  through  the  CT  image  slices  and  selecting  the  fiducials  using  the 
mouse.  For  paired-point  based  registration,  at  least  three  points  are  required  although  at  least  four  are  preferred. 

5.  Initialize  the  tracking  device. 

6.  Add  patient  landmarks  by  touching  the  physical  fiducials  attached  to  patient  using  the  tracked  pointer  device. 

7.  Perform  the  image  to  patient  landmark  registration. 

8.  Path  planning.  The  surgeon  will  select  a  target  point  and  an  entry  point  to  plan  the  path  for  the  needle  puncture. 

9.  Provide  a  real-time  display  of  the  overlay  of  the  needle  probe  and  pre-operative  images  in  the  quadrant  viewer 
window  during  the  biopsy  procedure. 

Since  the  application  is  implemented  as  a  state  machine,  the  completion  of  each  task  will  cause  the  application  to  enter  a 
new  state,  and  there  will  be  a  set  of  states  to  indicate  the  status  of  the  application.  The  user  interaction  with  the  GUI  can 
be  translated  into  inputs  to  the  state  machine.  For  instance,  when  we  click  on  the  register  patient  button,  this  will 
generate  a  “RequestSetPatientNamelnput”  to  the  state  machine.  The  state  machine  will  take  this  input  and  change  its 
current  state  from  “InitialState”  to  “WaitingForPatientName State”,  and  the  action  is  to  pop  up  a  window  asking  for 
input  of  the  patient  name.  If  the  user  inputs  a  valid  name,  then  there  will  be  a  “PatientNamelnpuf  ’  which  brings  the  state 
machine  into  “PatientNameReady State”,  otherwise  there  will  be  a  “PatientNameEmptylnput”,  which  will  return  the 
state  machine  to  the  “InitialState”.  Thus,  we  can  map  the  application  into  series  of  states  and  inputs  and  this  higher  level 
abstraction  will  help  the  developer  design  a  clear  work  flow  for  the  application.  Figure  2  shows  the  state  machine 
diagram  for  the  needle  biopsy  application  which  was  generated  automatically  when  the  state  machine  is  constructed 
using  the  dot  tool  from  graphviz  [10]. 
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Figure  2:  State  machine  diagram  for  needle  biopsy  application.  (Circle  for  state  and  arrow  for  transition  and  corresponding  input) 
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3.2  Coding  the  state  machine 

This  section  shows  how  to  code  the  state  machine  into  the  needle  biopsy  application.  IGSTK  has  a  number  of 
convenient  macros  to  facilitate  the  programming  of  the  state  machine.  The  details  of  these  macros  can  be  found  in 
Source/igstkMacro.h.  More  information  on  the  state  machine  design  pattern  and  guidelines  can  be  found  on  the  IGSTK 
wiki  page  under  “Development”  section  on  “Design  discussions”  page  [12].  Note  that  these  URLs  are  current  as  of  the 
time  of  publication  (Feb.  2006)  but  are  subject  to  change  in  the  future. 

Once  we  have  the  higher  level  abstraction  of  the  application  and  prototyped  it  in  the  state  machine  model,  we  need  to 
take  the  following  steps  to  program  the  state  machine  into  the  applications. 

1.  The  first  step  is  to  use  the  state  machine  declaration  macro  in  your  class’s  header  file.  This  macro  defines  types  for 
state  and  input,  creates  a  private  member  variable  mStateMachine,  and  two  private  member  functions  for  exporting  the 
state  machine  description  into  dot  format  for  the  state  machine  diagram  visualization  [10]  and  LTSA  (Labeled 
Transition  Systems  Analyzer)  format  for  state  machine  animation  and  validation  [11]. 

igstkStateMachineMacro(); 

2.  Then  take  the  states  and  inputs  mapped  out  during  the  prototyping  stage  and  define  them  in  the  header  file  using  the 
following  macros.  To  enforce  the  naming  conventions  of  the  state  machine,  the  declaration  macros  will  append  “State” 
or  “Input”  automatically  after  the  variable  name.  For  instance,  the  following  two  lines  will  define  mlnitialState  and 
mRequestLoadlmagelnput. 

igstkDeclareStateMacro(  Initial ); 
igstkDeclareInputMacro( RequestLoadlmage ); 

3.  The  next  step  is  to  construct  the  state  machine  in  the  constructor  of  the  source  file.  First,  we  need  to  add  all  the  states 
and  inputs  declared  in  the  header  into  the  state  machine. 

igstkAddStateMacro(  Initial ); 
igstkAddInputMacro(  RequestLoadlmage  ); 

4.  The  next  step  is  a  crucial  step  which  creates  the  state  machine  transition  table  to  control  the  logic  and  workflow  of  the 
application.  This  is  done  using  the  macro  igstkAddTransitionMacro(  FromState,  Receivedlnput,  ToState,  Action). 
This  means  when  the  state  machine  is  in  the  “From  State”  and  receives  the  “Received  lnput’ ’,  it  will  enter  into  the 
“To  State”  and  evoke  the  ActionProcessing()  as  an  action  for  this  transition.  This  macro  requires  the 
“ActionProcessing”  method  to  be  pre-defmed  in  the  class  for  the  state  machine  to  call.  For  example: 

igstkAddTransitionMacro( Initial,  RequestSetPatientName,  WaitingForPatientName,  SetPatientName  ); 

In  this  case,  we  need  to  have  a  SetPatientNameProcessing()  method  defined  in  the  class  for  this  code  to  compile. 

5.  After  we  have  setup  the  transition  table,  the  next  step  is  to  select  an  initial  state,  and  flag  the  state  machine  to  be 
ready  to  run.  After  the  state  machine  is  ready  to  run,  we  cannot  change  the  state  machine  transition  table  in  the  code. 
This  is  designed  this  way  to  enhance  the  safety  of  the  state  machine  and  prevent  accidentally  changes  to  the  state 
machine  behavior  in  the  code. 

igstkSetInitialStateMacro(  Initial ); 
m_StateMachine.SetReadyToRun() ; 

6.  Now  the  state  machine  is  setup  and  ready  to  run.  We  can  then  export  the  state  machine  description  in  dot  format  and 
generate  the  graphical  visualization  as  shown  in  Figure  2.  This  graph  will  help  us  to  examine  the  workflow  of  the 
application  and  the  state  transition  table. 

std::ofstream  ofile; 

ofde.  open("DemoApplicationStateMachineDiagram.  dot”); 
const  bool  skipLoops  =  false; 

this->ExportStateMachineDescription( ofile,  skipLoops  ); 
ofile. closeQ; 
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This  will  output  the  state  machine  into  a  dot  file  when  we  execute  the  application.  If  you  have  the  dot  tool  installed  in 
your  system,  then  you  can  run  the  following  command,  which  will  take  the  dot  file  and  generate  a  png  format  picture 
named  “SMDiagram.png”  for  the  state  machine. 

>dot  -T png  -o  SMDiagram.png  DemoApplicationStateMachineDiagram.dot 

7.  All  the  requests  to  a  state  machine  should  be  translated  into  inputs  and  the  state  machine  will  response  to  those  inputs 
depending  on  its  current  state.  These  actual  actions  should  be  protected  methods  and  only  called  by  the  state  machine 
directly.  In  the  code,  a  click  on  the  load  image  button  will  be  translated  to  a  ‘RequestLoadlmagelnput’ ,  and  then  we  call 
ProcessInputs()  to  let  the  state  machine  handle  this  request. 
igstkPushInputMacro(  RequestLoadlmage  ); 
m_StateMachine.ProcessInputs() ; 

If  the  state  machine  is  in  the  right  state  to  load  the  image,  a  protected  method  associated  with  this  transition  (eg. 
LoadImageProccessing()  )  will  be  evoked  by  the  state  machine  as  defined  in  the  transition  table  constructed  in  the 
constructor. 

3.3  Discussion 

From  the  computational  theory  point  of  view,  all  computers  are  state  machines,  and  all  computer  programs  are  state 
machines  regardless  of  whether  the  developers  used  the  state  machine  programming  pattern  or  not.  Traditional 
programming  approaches  represent  the  states  ambiguously  by  using  a  large  number  of  variables  and  flags,  which  result 
in  many  conditional  tests  in  the  code  (if-then-else  or  switchcase  statements  in  C/C++).  Programmers  could  neglect  to 
consider  all  possible  paths  in  the  code  while  struggling  with  if-else  conditional  tests  and  flag  checks.  These  practices 
may  result  in  unpredictable  behavior  and  limit  safety  in  the  design  of  the  underlying  applications.  Since  predictability  is 
critical  for  mission  critical  applications  running  in  the  surgery  room,  this  approach  is  not  suitable  for  our  purpose. 

In  comparison  to  traditional  approaches,  state  machines  will  reduce  the  number  of  paths  in  the  code,  save  the  developers 
from  convoluted  conditional  tests,  and  encourage  them  to  focus  on  higher  level  design.  From  the  above  example,  we 
can  see  that  the  state  machine  is  easy  to  program  and  manage  under  the  IGSTK  framework.  We  encourage  developers  to 
design  and  code  the  state  machine  of  their  application  first,  and  then  generate  the  state  machine  diagram  as  shown  in 
Figure  2.  They  can  go  through  the  diagram,  examine  and  verify  their  design  of  the  work  flow.  If  they  want  to  add  or 
change  a  path  of  the  application,  it  is  just  a  matter  of  adding  or  deleting  a  transition  table  entry.  This  eliminates  the  level 
of  difficulty  required  for  going  through  the  code  and  struggling  with  if-then-else  logic.  This  will  largely  facilitate  the 
application  prototyping,  and  the  implementation  code  can  be  plugged  into  the  skeleton  program  later.  These  techniques 
should  result  in  clearer  designs  and  safer  applications. 

Figure  3  shows  the  user  interface  of  the  needle  biopsy  application  written  in  FLTK.  The  left  side  is  the  control  panel, 
consists  of  a  set  of  buttons  corresponding  to  the  series  of  tasks  performed  during  the  procedure.  These  buttons’ 
callbacks  should  call  the  public  request  methods  of  the  application,  which  will  be  translated  into  state  machine  inputs. 
The  state  machine  will  then  take  proper  action  according  to  its  own  state.  For  example,  when  the  patient  information  is 
not  set,  the  ‘Load  Image’  button  won’t  respond  to  the  user  click.  There  is  no  need  for  conditional  checks  or  disabling  of 
buttons  here  as  these  actions  are  already  in  the  state  machine  transition  table.  On  the  right  hand  side,  there  are  four 
standardized  views,  axial,  sagittal,  coronal,  and  3D  view.  Here  we  loaded  an  abdominal  phantom  CT  image.  The  green 
cylinder  represents  the  needle  being  tracked  by  the  tracker.  The  viewer  will  automatically  reslice  the  images  as  the 
needle  tip  is  moving  in  the  anatomy. 
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Figure  3:  Snapshot  of  the  needle  biopsy  application 


4.  CONCLUSIONS 

The  major  components  of  the  toolkit  have  been  completed.  Automated  nightly  testing  results  can  be  found  at 
http://public.kitware.com/dashboard.php?name=igstk.  The  current  code  coverage  on  tests  is  around  95.7%.  A  state 
machine  validation  tool  is  being  developed  using  LTSA  [11],  which  can  animate  the  transition  of  the  state  machine. 
This  tool  can  be  found  on  the  IGSTK  wiki  page  under  “Quality”  section  on  “IGSTK  State  Machine  Validation”  [12]. 
This  tool  is  being  developed  to  test  all  possible  combination  of  inputs  to  a  state  machine,  which  will  allow  automated 
testing  of  applications  implemented  with  state  machine.  We  are  planning  on  releasing  three  demo  applications  with  this 
toolkit.  The  “Needle  Biopsy”  application  as  illustrated  in  the  previous  sections  is  available.  “Ultrasound  Guided  RFA” 
and  “Guidewire  Tracking”  will  be  available  before  this  phase  of  the  project  ends  later  this  year. 

The  IGSTK  toolkit  is  open  source  software  distributed  under  a  BSD-like  license.  Basic  information  related  to  IGSTK 
can  be  found  at  the  website  www.igstk.org  and  the  wiki  pages  [12].  Instructions  for  configuring  and  building  the  toolkit 
are  available  at  IGSTK  wiki  page  under  “Documentation”  section  on  “How  to  build  IGSTK”  and  “IGSTK:  Tutorial”. 
You  are  welcome  to  try  the  software,  review  the  source  code  and  send  your  comments  and  report  bugs  to  the  IGSTK 
user  list,  igstk-users@public.kitware.com.  Note  that  this  software  should  only  be  used  in  clinical  cases  under  IRB 
approval.  You  are  allowed  to  use  IGSTK  for  free  in  academic  and  commercial  applications  but  it  is  your  responsibility 
to  perform  the  tests  and  validations  required  by  regulatory  bodies  such  as  the  U.S.  Food  and  Drug  Administration 
(FDA). 
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Abstract 

Many  different  robotic  systems  have  been  developed  for  invasive  medical  procedures.  In 
this  article  we  will  focus  on  robotic  systems  for  image-guided  interventions  such  as 
biopsy  of  suspicious  lesions,  interstitial  tumor  treatment,  or  needle  placement  for  spinal 
blocks  and  neurolysis.  Medical  robotics  is  a  young  and  evolving  field  and  the  ultimate 
role  of  these  systems  has  yet  to  be  determined.  This  paper  presents  four  interventional 
robotics  systems  designed  to  work  with  MRI,  CT,  fluoroscopy,  and  ultrasound  imaging 
devices.  The  details  of  each  system  are  given  along  with  any  phantom,  animal,  or 
human  trials.  The  systems  include  the  AcuBot  for  active  needle  insertion  under  CT  or 
fluoroscopy,  the  B-Rob  systems  for  needle  placement  using  CT  or  ultrasound,  the 
INNOMOTION  for  MRI  and  CT  interventions,  and  the  MRBot  for  MRI  procedures. 
Following  these  descriptions,  the  technology  issues  of  imager  compatibility,  registration, 
patient  movement  and  respiration,  force  feedback,  and  control  mode  are  briefly 
discussed.  It  is  our  belief  that  robotic  systems  will  be  an  important  part  of  future 
interventions,  but  more  research  and  clinical  trials  are  needed.  The  possibility  of 
performing  new  clinical  procedures  that  the  human  cannot  achieve  remains  an  ultimate 
goal  for  medical  robotics.  Engineers  and  physicians  should  work  together  to  create  and 
validate  these  systems  for  the  benefits  of  patients  everywhere. 
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1  Introduction 

Many  different  robotic  systems  have  been  developed  for  invasive  medical  procedures.  In 
this  article  we  will  focus  on  robotic  systems  for  minimally  invasive  interventions  such  as 
biopsy  of  suspicious  lesions  or  needle  placement  for  spinal  blocks.  According  to  the 
Robotic  Institute  of  America,  a  robot  is  "a  reprogrammable,  multifunctional  manipulator 
designed  to  move  materials,  parts,  tools,  or  other  specialized  devices  through  various 
programmed  motions  for  the  performance  of  a  variety  of  tasks."  For  our  purposes,  this 
definition  fits  most  medical  robotics  systems  fairly  well,  in  that  they  are  typically 
mechanical  manipulators  with  rigid  links  connected  by  joints  that  allow  relative  motion 
from  one  link  to  another  (1 ). 

Medical  robotics  is  a  relatively  young  field,  with  the  first  recorded  medical  application  of  a 
robot  occurring  in  1985  (2).  Unlike  factory  robotics,  in  which  virtually  all  operations  are 
now  automated  and  the  use  of  robotics  is  widespread,  medical  robotics  is  still  a  niche 
field.  While  medical  robots  have  been  applied  in  many  fields  such  as  neurosurgery, 
orthopedics,  and  urology,  they  are  not  the  standard  of  care  in  any  field  and  in  fact  very 
limited  market  penetration  has  occurred.  This  limited  use  is  no  doubt  due  to  the  many 
challenges  that  need  to  be  overcome  in  developing  a  robotic  system  for  a  medical 
application.  In  particular,  safety  is  an  overriding  concern  and  must  be  considered  from 
the  start  in  any  medical  robotic  system.  Note  that  in  the  factory,  we  do  everything  we  can 
to  keep  people  away  from  robots,  but  in  the  medical  environment  many  robotic  systems 
are  designed  to  work  with  people  nearby,  and  all  medical  robotics  systems  must  ensure 
safe  operation  with  a  patient  in  the  workspace  of  the  robot.  Despite  these  challenges,  we 
believe  that  medical  robotic  systems  have  a  place  in  minimally  invasive  procedures  and 
this  article  describes  several  systems  developed  by  the  authors  for  this  purpose. 

There  are  many  other  robotic  systems  which  have  been  developed  and  there  are  several 
review  articles  that  have  been  written  in  the  past  few  years  such  as  (3-6)  In  particular, 
there  are  two  systems  that  are  currently  used  in  clinical  practice  that  should  be 
mentioned  here.  The  first  system  is  the  da  Vinci®  from  Intuitive  Surgical  (Sunnyvale, 
California,  USA)  which  functions  along  the  line  of  a  master  -  slave  telemanipulator  for 
endoscopic  surgical  procedures  (7).  The  da  Vinci  consists  of  the  surgeon’s  viewing  and 
control  console,  a  control  unit,  and  a  three-arm  surgical  manipulator  (8).  While  the  initial 
application  of  the  system  was  cardiac  surgery,  a  recent  focus  of  the  company  has  been 
on  urological  surgery,  specifically  minimally  invasive  prostatectomy.  According  to  the 
manufacturer,  about  400  systems  have  been  installed  to  date.  The  second  system  is  the 
CyberKnife®  from  Accuray  (Sunnyvale,  California,  USA)  for  stereotactic  radiosurgery. 
The  CyberKnife  consists  of  a  lightweight  linear  accelerator,  a  KUKA  robot,  paired 
orthogonal  x-ray  imagers,  and  a  treatment  couch  (9).  The  system  was  originally 
developed  to  treat  tumors  in  the  brain  and  spine,  but  is  now  FDA  approved  to  treat 
lesions  anywhere  in  the  body  including  the  lung  and  pancreas.  According  to  the 
manufacturer,  over  100  systems  have  been  sold  to  date. 

The  article  is  organized  as  follows.  In  Section  2,  which  is  the  bulk  of  the  article,  we 
describe  four  interventional  systems  designed  to  work  with  MRI,  CT,  fluoroscopy,  and 
ultrasound  imaging  devices.  The  details  of  each  system  are  given  along  with  any 
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phantom,  animal,  or  human  trials.  The  paper  concludes  with  a  presentation  of  some 
technology  issues  in  Section  3  and  a  discussion  and  summary  in  Section  4. 


2  Interventional  Robotics  Systems 

In  this  section,  four  interventional  robotics  systems  which  were  developed  by  the  authors 
will  be  presented.  Table  1  lists  the  system  name,  institution  where  the  system  was 
developed,  the  status  (phantom,  animals,  clinical  trial),  and  the  imaging  modality  used. 
Note  that  these  systems  are  representative  examples  of  the  state  of  the  art  in 
interventional  robotics.  There  are  other  systems  that  have  been  built  and  the  reader  is 
referred  to  the  review  articles  listed  earlier  for  more  details. 


System 

Institution 

Status 

Imaging 

Modality 

Section  in 

paper 

AcuBot 

Hopkins/Georgetown 

(USA) 

Cadaver  studies 

Animal  studies 

Clinical  trial  done 

Fluoroscopy  and 

CT 

2.1 

B-Rob 

ARC  Seibersdorf  research 

(Austria) 

Phantom  studies 

CT  and 

ultrasound 

2.2 

INNOMOTION 

Innomedic/FZK/  FH  Ge 

(Germany) 

Animal  studies 

Clinical  use  starting 

CE  marked 

MRI  and  CT 

2.3 

MRBot 

Hopkins  (USA) 

Phantom  studies 

MRI 

2.4 

Table  1:  The  four  interventional  robot  systems  described  in  this  article 
2.1  Acubot 

The  AcuBot  (10)  robot  presents  a  modular  structure  of  several  functionally  distinctive 
components  that  have  been  developed  over  the  past  five  years  in  the  URobotics 
Laboratory  at  Johns  Hopkins  Medical  Institutions  (Baltimore,  USA).  The  AcuBot 
incorporates  the  original  PAKY  (percutaneous  access  of  the  kidney)  radiolucent  needle 
driver  (1 1);  a  RCM  (remote  center  of  motion)  module  capable  of  needle  orientation  (12); 
an  XYZ  Cartesian  stage  for  translational  positioning  of  the  needle  tip;  and  a  passive 
positioning  arm  (S-arm)  mounted  onto  a  bridge  frame. 
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Figure  1  shows  two  views  of  the  robot  mounted  on  a  CT  scanner.  The  base  of  the  robot 
[2]  provides  a  bridge-like  structure  over  the  table.  The  robot  has  a  total  of  six  active 
degrees  of  freedom  (DOF)  configured  for  decoupled  positioning,  orientation,  and 
instrument  insertion. 


Figure  1 :  a)  Front  and  b)  side  views  of  the  AcuBot 


The  instrument  [7]  is  loaded  in  PAKY  [6],  which  is  an  active  radiolucent  needle  driver  (T 
translation).  PAKY  is  held  by  the  ROM  (11)  module  [5].  This  module  is  capable  of 
precisely  orienting  the  instrument  about  two  nearly  perpendicular  directions  (Rx  and  Rz) 
coincident  at  the  ROM  point,  thus  allowing  a  pivoting  motion  about  that  point.  The  RCM 
is  supported  by  a  passive  positioning  arm  [4],  called  the  S-ARM,  with  7  DOF  (SI 
spherical,  R  revolute,  S2  spherical).  The  arm  can  be  positioned  and  rigidly  locked  from  a 
single  lever.  The  base  of  the  arm  [4]  is  mounted  in  a  3  DOF  Cartesian  stage  [3],  the  XYZ 
module  (Tx,  Ty,  and  Tz  translations).  The  user  interface  consists  of  a  15”  resistive  touch 
screen  [8],  a  two-axis  joystick,  a  switch  panel  [10],  and  an  emergency  stop  button  [11]. 
These  components  are  mounted  on  the  front  side  of  the  bridge.  A  speaker  [9]  and  the 
cable  connector  [12]  are  located  on  its  back  side. 


2.1.1  Needle  Registration  and  Clinical  Studies  in  CT  Scanners 

The  method  is  based  on  aligning  the  procedure  needle  held  by  the  robot  with  the  laser 
markers  of  the  CT  scanner.  The  robot  can  then  automatically  orient  the  needle  toward  a 
target  selected  in  a  CT  slice.  The  skin  entry  point  and  target  location  may  be  contained 
in  different  slices.  Needle  insertion  may  then  be  performed  under  joystick  control  or 
automatically  under  CT  fluoroscopy  (CTF)  monitoring  by  the  physician. 

This  method  showed  an  experimental  accuracy  of  less  than  1  mm  in-slice  and  1 .5  mm 
for  out-of-slice  targets.  Four  clinical  cases  of  kidney  and  spine  biopsy  and  RF  ablation 
(Figure  2),  and  a  nephrostomy  tube  placement  were  successfully  performed  with  no 
complications  (13).  These  cases  demonstrated  a  significant  improvement  over  the 
manual  method,  in  which  the  needle  is  typically  restricted  to  the  CTF  plane  so  that  the 
needle  can  always  be  seen  by  the  physician. 
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Another  study  (14)  with  the  same  robotic  system  found  the  in-vitro  accuracy  of  the  robot 
to  be  0.6°  angular  and  1.65  mm  linear.  The  clinical  study  included  10  percutaneous  core 
biopsies  (7  kidney,  2  lung,  1  liver),  1 1  RF  ablations  (9  kidney,  2  spine),  1  nephrostomy 
tube  placement,  and  one  neobladder  access.  In  four  cases,  the  target  was  not  met 
adequately,  and  fine-tuning  adjustment  with  the  joystick  was  required  to  reach  the  target. 
In  all  cases,  however,  the  study  showed  that  the  use  of  the  robot  reduced  radiation 
exposure  for  the  patient  and  medical  personnel. 


Figure  2:  CT-guided  RF  ablation  with  PAKY-RCM  system  and  laser  based  registration 


2.1.2  Clinical  Trial  for  Nerve  Blocks  under  Fluoroscopy 

After  cadaver  studies  using  the  robot  to  precisely  position  a  needle  in  the  lumbar  spine 
were  successfully  completed  (15)  in  the  Department  of  Radiology  at  Georgetown 
University,  a  randomized  clinical  trial  of  20  patients  undergoing  nerve  and  facet  blocks 
was  approved  by  the  FDA  and  the  local  institutional  review  board.  The  procedure  was 
done  following  the  usual  clinical  practice  except  the  robot  was  used  to  position,  orient, 
and  drive  the  needle  under  physician  control.  A/P  fluoroscopy  was  used  to  position  and 
orient  the  needle,  and  lateral  fluoroscopy  was  used  to  monitor  the  depth  of  insertion. 

The  robot  was  mounted  on  the  interventional  table  using  a  custom-designed  locking 
mechanism.  The  robot  was  positioned  initially  near  the  skin  entry  point  by  loosening  the 
passive  gross  positioning  mechanism  and  moving  the  needle  driver  end  of  the  robot  by 
hand.  Once  this  initial  position  had  been  attained,  the  mechanism  was  locked  and  the 
robot  was  switched  to  operate  by  physician  control  using  the  joystick. 

The  study  was  completed  by  a  single  fellowship  trained  interventional  neuroradiologist  at 
Georgetown  University  Hospital  using  a  Siemens  Neurostar  bi-plane  fluoroscopy 
system.  The  standard  manual  technique  was  used  on  10  patients  and  the  robotic  device 
was  used  on  10  patients.  The  patients  ranged  in  age  from  30  to  70  years.  The  spine 
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levels  were  from  S-1  to  L-5.  No  complications  were  observed  in  the  study.  One  of  the 
patients  in  the  robotics  arm  had  to  be  converted  to  a  manual  procedure  due  to  slippage 
of  the  needle  driver.  This  conversion  was  done  without  difficulty  or  complications. 

There  were  two  outcome  measures:  1)  accuracy  of  needle  placement  and  2)  pain  relief. 
Accuracy  of  needle  placement  was  determined  as  follows.  Before  the  interventionalist 
began  placing  the  needle,  both  an  A/P  and  lateral  image  of  the  patient  were  obtained. 
The  interventionalist  would  then  annotate  each  image  with  an  arrow  to  indicate  the 
desired  target  location  of  the  needle  (the  interventionalist  was  not  blinded  as  to 
manual/robotic  technique  as  this  was  not  practical).  After  the  needle  was  placed,  an  A/P 
and  lateral  image  was  again  obtained.  The  two  sets  of  images  were  compared  to 
determine  the  distance  between  the  intended  location  of  the  needle  and  the  actual 
location  of  the  needle.  Pain  relief  was  measured  using  a  visual-analog  scale,  with  0 
representing  no  pain  and  10  representing  excruciating  pain. 

The  results  to  date  show  that  is  feasible  to  use  a  joystick  controlled  robot  for  nerve  and 
facet  blocks.  An  example  case  is  shown  in  Figure  3.  While  this  was  a  pilot  study  and  not 
enough  data  was  gathered  for  statistical  significance,  some  general  trends  can  be 
observed.  The  mean  accuracy  in  the  robot  (1.105  mm)  and  manual  (1.238  mm)  is  about 
the  same.  Therefore,  it  appears  that  the  robot  is  capable  of  accurate  needle  placement. 

As  expected,  the  pain  score  post-treatment  was  significantly  less  than  the  pain  score 
pre-treatment  in  both  the  robot  and  manual  arms.  In  the  robot  arm,  pain  scores  fell  from 
a  mean  of  6.3  pre-treatment  to  1 .8  post-treatment.  In  the  manual  arm,  pain  scores  fell 
from  6.0  pre-treatment  to  0.9  post-treatment.  Patients  had  to  sign  an  informed  consent 
form  and  were  generally  receptive  to  the  use  of  the  robot. 


Figure  3:  Clinical  trial  of  robotically  assisted  nerve  blocks 
at  Georgetown  University  under  biplane  fluoroscopy 
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2.2  B-Rob  Systems  -  ARC  Seibersdorf  Research 

Robotic  systems  for  CT  and  ultrasound-guided  biopsies  have  been  developed  by  the 
robotics  laboratory  of  ARC  Seibersdorf  Research  in  Austria.  The  two  systems  developed 
by  this  group  are  presented  here. 

2.2.1  Prototype  Biopsy  Robot  I  (B-Robl) 

The  first  prototype  biopsy  robot  was  called  B-Robl  and  was  a  seven  degree  of  freedom 
(DOF)  stand-alone  robot  system  integrated  on  a  mobile  rack  (16).  The  biopsy  instrument 
is  positioned  at  the  skin  entry  point  by  a  4-DOF  gross  positioning  system  consisting  of 
three  Cartesian  linear  axes  together  with  one  additional  rotational  link  for  a  rough 
orientation  of  the  needle.  For  final  orientation  of  the  needle  the  robot  is  equipped  with  a 
“Needle  Positioning  Unit”  (NPU)  consisting  of  two  linear  DOFs  which  move  two  parallel 
carbon  “fingers”  connected  by  spherical  links.  Another  linear  DOF  with  a  limited  stroke  of 
50mm  can  move  the  entire  NPU  toward  the  skin  entry  point  in  a  safe  approach 
movement,  i.e.,  with  minimal  velocity  and  force.  The  needle  orientation  stage  is  thus 
strictly  decoupled  from  movement  of  any  axis  of  the  gross  positioning  system.  A  remote 
center  of  motion  (“pivot  point”)  for  angulation  of  the  needle  is  maintained  by  the 
kinematic  structure  of  the  NPU  as  another  safety  measure  during  the  intervention. 

The  robot  system  is  controlled  by  two  industrial  PCs.  One  PC  provides  high-level  control 
of  the  robot  system  and  a  second  PC  handles  the  interface  to  the  optical  tracker  system 
(Polaris,  Northern  Digital)  as  well  as  the  planning  and  monitoring  software.  This  PC  also 
includes  a  video  capture  card  (WinTV-PCI-FM  718,  Hauppauge)  for  grabbing  images 
from  an  ultrasound  probe  or  the  CT  monitor  to  support  planning  of  an  intervention.  After 
acquisition  of  images  of  the  target  region,  the  physician  selects  the  desired  skin  entry 
point  as  well  as  the  target  point.  With  that  information  the  relevant  data  (angulation, 
distance  to  the  target  lesion)  are  calculated  and  automatically  sent  to  the  robot  controller 
via  TCP/IP  socket  connection.  Using  the  GUI  of  the  planning  software,  the  virtual 
trajectory  of  the  biopsy  can  be  viewed  in  all  CT-slices  involved  to  verify  the  intervention 
path.  After  planning  of  the  intervention,  the  robot  can  be  moved  towards  its  final  position 
by  a  coordinated  motion  of  the  axes.  The  gross  positioning  unit  can  then  be  locked  if 
desired.  The  NPU  is  then  moved  to  the  skin  entry  point  under  very  controlled  conditions 
(at  low  speed  and  with  limited  distance)  and  the  needle  can  be  manually  inserted. 

The  performance  of  the  complete  system  has  been  extensively  evaluated  in  a  series  of 
in  vitro  tests  using  a  needle-penetrable  phantom  (17-19).  Peas  (mean 
diameter=9.4±0.7mm)  were  embedded  as  targets  within  a  custom-made  gel-phantom. 
Based  on  the  intervention  plan,  the  NPU  was  commanded  to  the  desired  skin  entry  point 
to  provide  guidance  for  a  17-gauge  coaxial  puncture  needle  and  an  18-gauge  long 
biopsy  needle.  After  manual  needle  insertion,  sample  harvesting  was  performed  by 
means  of  an  automated  biopsy  device  (Magnum  Core  high  speed;  22-mm  excursion). 
The  distance  between  the  actual  needle  tract  and  the  centre  of  the  target  was  evaluated 
in  two  orthogonal  axes  using  ultra-sonography  -  the  length  of  the  harvested  biopsy 
specimen  also  was  evaluated  by  direct  measurement.  Test  series  were  performed  for 
both  US-guided  biopsies  (scanning  head  C4-2;  US-System  HDI-UM  9,  Advanced 
Technology  Laboratories,  USA)  as  well  as  for  CT-guided  interventions  (Multidetector  CT, 
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Siemens  Somatom  Sensation  16;  CareVision  Mode,  0.75  c  0.75,  80kV,  160  mAs,  0.5 
sec).  Photographs  of  the  CT  tests  are  shown  in  Figure  4  and  Figure  5. 

The  system  showed  sufficient  operational  stability  and  accuracy  for  the  procedures 
under  consideration.  The  measured  targeting  accuracy  (1 ,48mm  ±  0.62mm)  is  better 
compared  to  traditional  techniques  by  additionally  combining  the  advantages  of  needle 
guidance  and  free-hand  technique.  Integration  of  the  complete  system  on  a  mobile  rack 
allows  short  setup  time  and  easy  installation  of  the  system  at  different  sites.  On  the  other 
hand,  the  chosen  approach  leads  to  a  very  bulky  system  and  to  a  very  high  grade  of 
automation. 


Figure  4:  Biopsy  robot  I  during  in  vitro  testing  for 
image-guided  interventions  under  CT  fluoroscopy 
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Figure  5:  Close-up  of  testing  showing  phantom  (peas  can  be  seen  in  the  bag), 
needle  positioning  unit  (NPU),  and  optical  trackers 


2.2.2  Biopsy  Robot  II  (B-Robll) 

The  main  goal  for  this  new  design  was  to  transfer  the  concepts  demonstrated  from  the 
B-Robl  prototype  into  a  practical  clinical  setup.  The  major  goals  for  the  new  development 
were: 

1 .  a  modular  setup  for  a  broad  variety  of  clinical  applications, 

2.  a  significant  reduction  of  technical  complexity  (compared  to  the  previous 
prototype)  to  reach  an  acceptable  cost/benefit  ratio  for  the  entire  system, 

3.  easy  integration  to  devices  used  in  interventional  radiology, 

4.  seamless  integration  to  clinical  workflow,  and 

5.  a  “plug&play”  philosophy. 

The  mechanical  architecture  for  the  new  design  was  based  on  the  parallelogram 
mechanism  already  realized  for  the  NPU  of  the  B-Robl  prototype.  For  easy  sterilization, 
the  two  carbon  “fingers”  -  together  with  the  polymer  bearings  and  the  needle  guideway  - 
can  be  disconnected  from  the  positioning  module  (i.e.  the  robot)  by  means  of  a  rapid- 
change  bayonet  connection.  The  mechanical  design  of  the  device  is  low-profile 
(dimensions  of  one  2DOF  module:  WxLxH  =  100mm  x  150mm  x  30mm)  in  order  to  use 
the  system  inside  of  the  CT  gantry  without  major  restrictions.  Following  the  general  idea 
of  modularity,  different  configurations  are  supported  in  order  to  allow  simple  2DOF 
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needle  angulation  (+/-  30°),  2DOF  positioning  (+/-  20  mm)  as  well  as  (optional) 
maintaining  a  software  defined  pivot  point  for  angulation.  For  gross  positioning  of  the 
needle  entry  point,  the  module(s)  are  mounted  on  one  or  two  passive  7DOF  multi¬ 
functional  holding  arm(s)  (ATLAS  arm,  Medical  Intelligence  GmbH,  Schwabmiinchen, 
Germany). 

The  robot  control  system  was  developed  in-house  and  consists  of  two  axis  controllers  for 
each  module,  a  safety  card  which  disconnects  all  motors  from  the  power  supply  in  case 
of  an  emergency  stop,  and  a  power  supply  module  -  all  of  which  are  integrated  into  a 
HF-dense  19”  housing.  All  modules  are  interconnected  via  a  standard  RS485  bus 
system.  Operation  of  the  entire  system  is  synchronized  either  via  a  PC  (MDD  certified 
computer  or  panel  PC)  connected  to  the  RS485  bus  or  via  a  hand-held  control  unit  in 
stand-alone  operation.  Thanks  to  this  modular  setup,  single  modules  can  be  easily 
replaced  and  the  system  can  be  expanded  easily. 

Planning  of  the  intervention  is  based  on  imaging  data  sets  acquired  immediately  before 
an  intervention.  The  spatial  relation  between  the  imaging  space  and  the  targeting  device 
is  either  established  by  means  of  a  tracker  system  (optical  or  mechanical)  or  via  robot 
registration  based  on  a  CT-data  set.  After  graphical  selection  of  the  target  and  manual 
pre-positioning  of  the  device,  the  correct  angulation  will  be  set  automatically  by  the 
system.  During  the  intervention,  the  robotic  kinematics  holds  the  needle  guide  in  a 
predefined  position  and  orientation.  However,  the  insertion  of  the  needle  itself  will  be 
performed  manually  by  the  physician. 

The  first  in  vitro  trials  of  the  system  using  a  penetrable  gel  phantom  show  that  B-Robll 
allows  image-guided  positioning  of  a  biopsy  needle  with  high  accuracy  (0.66mm  ± 
0.27mm).  The  system  is  easy  to  use  and  does  not  considerably  interfere  with  the  clinical 
work-flow.  A  risk  analysis  of  the  complete  system  (20)  did  not  find  any  major  risks.  A 
series  of  quantitative  evaluation  studies  -  for  both  US  and  CT  guided  biopsies  and  for 
different  system  setups  (no/  mechanical/  optical  tracker;  remote  controlled  operation)  is 
currently  in  process.  Beside  biopsy  procedures,  further  clinical  applications  are  currently 
under  evaluation  at  different  research  centers.  The  long-term  goal  of  this  work  is  to 
create  a  multi-purpose  system  for  a  broad  range  of  percutaneous  treatments,  in  any  part 
of  the  body,  using  any  kind  of  intra-operative  image  guidance. 


Figure  6:  Biopsy  robot  II  in  a  2  by  2  degree  of  freedom  configuration 
mounted  on  two  passive  holding  arms 
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Figure  7:  Biopsy  robot  II  during  initial  phantom  study 


2.3  CT  and  MR-Compatible  Robotic  Instrument  Guiding  System  INNOMOTION 

MR-guided  percutaneous  interventions  have  been  clinically  established  with  open  low 
field  MR  systems  (21).  As  the  imaging  quality  of  closed  bore  scanners  is  superior  to 
open  field  system  but  the  access  to  the  patient  is  limited  a  fully  MR-compatible 
assistance  system  INNOMOTION  (Innomedic,  Herxheim  &  FZK  Karlsruhe  Germany) 
{Gutmann,  2002  #47;Bock,  2004  #48})  was  developed  to  provide  precise  and 
reproducible  instrument  positioning  inside  the  magnet.  MRI  compatibility  has  been 
achieved  through  testing  of  all  components  and  the  complete  system  in  different  field 
strength  magnets  including  a  1.5  T  Siemens  Magnetom  Symphony  and  Philips  1.0  T 
Gyroscan  and  1 .5  T  magnets.  Targeting  precision  has  been  determined  with  a 
mechanical  FARO  arm  on  ex  vivo  organ  models  embedded  in  agarose  gel.  Targeting 
precision  has  also  been  evaluated  during  MRI  guided  percutaneous  interventions  in  a 
porcine  animal  model.  The  system  is  shown  in  Figure  8. 

2.3.1  Technology 

The  pneumatic  robotic  assistance  system  is  fully  MR-compatible  and  consists  of  a  robot 
arm  which  can  be  manipulated  in  6  degrees  of  freedom  (22-23).  The  robot  arm  is 
attached  to  a  260°  arch  that  is  mounted  to  the  patient  table  of  the  scanner  and  can  be 
passively  prepositioned  on  either  side  of  the  arch  at  0°,  30°  and  60°  to  the  vertical 


Page  52 


Minimally  Invasive  Therapy  and  Allied  Technology,  Volume  15,  Issue  2 


according  to  the  region  of  interest  (e.g.  spine,  liver,  kidney,  breast).  Active  positioning 
measurements  are  achieved  via  fiber  optically  coupled  limit  switches,  along  with 
rotational  and  linear  incremental  sensors.  The  kinematics  of  the  device  has  been 
carefully  optimized  for  use  in  close  bore  MRI  scanners  and  the  CT  gantry.  Piezoelectric 
drives  were  tested  but  due  to  the  RF  noise  during  MRI  scanning  and  the  risk  of  inductive 
heating  of  the  electric  power  lines  they  were  not  used  and  pneumatic  cylinders  with  slow 
motion  control  have  been  developed  instead  to  drive  all  six  degrees  of  freedom. 

A  module  for  application  of  coaxial  probes  (e.g.  cannulae  for  biopsies,  RF  or  Laser 
Probe,  endoscopes,  etc.)  provides  two  degrees  of  freedom  in  X  and  Z  axes  and  is 
attached  to  a  robotic  arm  with  4  degrees  of  freedom.  This  design  assures  stable 
positioning  of  the  instrument  with  in  a  tool  center  point  that  keeps  the  “invariant  point  of 
insertion”  at  the  skin  entry  point. 


Figure  8:  INNOMOTION  robot  for  MRI-guided  procedures 

The  application  module  (Figure  9)  for  clinical  use  provides  manual  translation  and 
rotation  of  the  cannula.  A  pneumatic  drive  has  been  developed  to  insert  the  cannula  in 
incremental  steps  of  1  -  20  mm.  In  conjunction  with  the  two  axes  for  movement  about 
the  tool  center  point  (+/-  30°)  the  instrument  trajectory  can  be  changed  to  other  targets 
without  moving  the  robot  arm  or  repositioning  the  arm  on  the  arch.  The  arch  is  movable 
and  can  be  firmly  attached  to  the  patient  table  of  the  MR  system  with  exchangeable 
fittings  easily  adapted  to  other  MRI  platforms  (Figure  10).  A  graphical  user  interface 
provides  trajectory  planning  directly  on  the  MRI  images  (Figure  1 1  e). 


Figure  9:  Application  module  for  manual  cannula  insertion 
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Figure  10:  INNOMOTION  with  six  degrees  of  freedom  can  be  mounted  on  different  types  of 

MRI  patient  beds 


Figure  11:  Equipment  set  up  (a)  phantom,  (b)  input  panel  and  (c)  in  room  control  unit ,  (d) 
pneumatic  control,  (e)  graphical  user  interface  and  (f)  MRI  monitor 


2.3.2  Procedure  Technique 

The  patient  is  placed  in  a  predetermined  position  suitable  for  the  intervention  (supine, 
prone  or  lateral).  The  system  is  prepositioned  and  firmly  attached  to  the  table  with 
clamps.  Based  on  the  pre-interventional  images  and  the  anatomical  region  of  interest  the 
table  is  moved  using  the  projection  of  the  laser  lights  from  the  MRI  gantry.  The  robot  is 
referenced  to  the  coordinate  system  of  the  MR  scanner  using  the  same  laser  line.  The 
arm  moves  back  and  forth  and  returns  so  that  the  light  detectors  at  the  upper  part  of  the 
application  module  are  aligned  with  the  laser  (within  +/-  0.5  mm).  The  laser  light  is 
switched  off  and  the  table  is  automatically  moved  in  to  the  MRI  bore  until  the  position  of 
the  laser  line  matches  with  the  zero  position  of  the  Z-axis  of  the  MRI  scanner.  Planning 
for  the  intervention  is  performed  by  using  fast  gradient  echo  sequences  in  transverse, 
sagittal  or  coronal  orientation.  Suitable  slices  are  selected  and  sent  via  the  network  in 
DICOM  format  to  the  computer  of  the  robotic  assist  system.  The  insertion  site  and  a 
target  point  are  selected  on  the  graphical  user  interface  and  the  corresponding 
coordinates  are  sent  to  the  control  unit.  The  drives  are  activated  and  the  application 
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module  is  moved  with  the  tool  center  point  to  the  insertion  site  on  the  skin.  The  cannula 
can  then  be  inserted  through  a  guiding  sleeve  or  along  an  open  angle. 


2.3.3  Evaluation  of  Targeting  Precision 

Mechanical  targeting  precision  has  been  determined  with  a  FARO  arm  under  dry  lab 
conditions.  The  MRI  procedures  were  performed  on  1 .5  T  Siemens  Magnetom 
Symphony,  and  a  Philips  1.0  T  Gyroscan  and  1.5  T  Intera.  The  test  were  done  on  ex 
vivo  organ  models  which  consisted  of  fresh  porcine  kidney  embedded  in  Agarose  and 
Gelatine  (Figure  12). 


Figure  12:  MRI  Target  precision  of  20  Gauge  cannula  insertion  into  a  porcine  kidney 

embedded  in  Agarose 


Targeting  precision  was  also  evaluated  during  MRI-guided  percutaneous  interventions  in 
an  porcine  animal  model  under  general  anesthesia  (Isoflurane).  The  animals  (four  3- 
month  old  domestic  pigs  30  -  40  kg)  were  placed  prone  on  the  patient  table  and  a 
surface  coil  was  fixed  around  the  planed  insertion  site  lateral  to  the  spine.  Using  T1- 
and  T2-weighted  planning  images,  the  appropriate  region  of  interest  was  defined  on  the 
graphical  user  interface  of  the  INNOMOTION  control  computer  (Figure  13).  The  robot 
arm  then  moved  and  oriented  the  needle  holder  to  the  insertion  point  automatically.  20 
and  22  gauge  MR-compatible  Titanium  grade  4  cannulae  (MRI  Devices-Daum, 

Schwerin,  Germany)  were  then  manually  inserted.  Subsequent  to  an  initial  insertion  of 
about  10  mm  the  table  was  repositioned  in  the  MRI  bore  and  control  images  were 
acquired.  The  precision  of  the  insertion  point  and  the  insertion  angle  were  determined  by 
overlaying  the  pre-interventional  images  with  the  new  MRI  image. 
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Figure  13:  Overlay  images  on  the  INNOMOTION  screen  for  evaluation  of  target  precision 


The  intervention  was  completed  within  the  magnet  from  the  rear  opening,  where  an  MR- 
compatible  in-room  monitor  was  placed.  During  the  insertion  of  the  needle,  real-time  MR 
images  were  acquired  for  guidance.  To  visualize  the  advancement  of  the  cannula  in  the 
tissue  fast  Gradient  Echoes  sequences  (TR  =  4.4  ms;  TE  =  2.2  ms;  FA  70°;  TA  =  0.7  s) 
were  used.  At  the  desired  region  of  interest  (nerve  root,  plexus  coeliacus)  spin  echo 
images  were  acquired  for  verification  of  the  cannula  position  through  a  test  bolus  of 
contrast  agent  solution  (NaCI/GdDTPA:  100/litre  of  0.8%  saline  solution).  The  injection 
was  done  under  real-time  MRI  (TR  =  1 .8  ms;  TE  =  4.3  ms  ;  TA  =  0.5  -  0.8sec.;  FA  =  20°) 
to  visualize  the  drug  distribution.  Final  therapeutic  injection  of  10-25  ml  with  contrast 
dotted  Mepivacainhydrochlorid  (Scandicain®1%,  Astra  -  Zeneca,  Germany)  was 
performed. 

2.3.4  Results  and  Discussion 

All  procedures  were  completed  successfully  including  injections  at  the  sympathetic 
chain,  sciatic  nerve  and  coeliac  plexus.  The  direct  MRI  control  and  new  sequences 
techniques  allows  correction  of  the  insertion  path  in  case  of  misdirection  due  to 
anatomical  structures.  The  insertion  site  and  the  insertion  angle  have  been  evaluated  by 
manual  measurement  on  overlays  of  the  planning  image  of  INNOMOTION  and  the 
subsequent  MR  control  image  (Fig.  13).  The  position  and  orientation  of  all  cannula 
insertions  were  appropriately  visualized  on  axial  MRI  images.  The  precision  of  the 
insertion  site  in  the  axial  plane  was  +/- 1  mm  (minimum  of  0.5  mm  and  maximum  of  3 
mm).  The  angular  deviation  in  the  transverse  plane  of  the  cannulae  was  +/- 1°  with  a 
minimum  of  0.5°  and  maximum  of  3°. 

Cross  platform  MRI  compatibility  can  be  achieved  by  using  polymers,  ceramics, 
pneumatic  drives  and  optoelectronic  sensors.  For  MRI-guided  cannula  interventions,  as 
the  cannula  is  currently  advanced  manually,  the  access  is  difficult  if  the  insertion  is  done 
inside  the  magnet.  Therefore,  the  direct  control  of  the  insertion  under  real-time  MRI  is 
recommended  to  allow  correcting  the  insertion  in  case  of  misdirection  of  the  cannula  and 
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to  precisely  position  the  tip  of  the  cannula  in  the  volume  of  interest.  To  ease  the 
procedure  tip  tracking  techniques  haven  been  evaluated  (23). 

2.4  MrBot:  A  Fully  MRI  Compatible  Robot  for  Prostate  Image-Guided 
Interventions 

A  new  robot,  MrBot  (24),  has  been  recently  developed  at  Hopkins  for  fully-automated 
image-guided  access  of  the  prostate  gland.  The  robot  is  customized  for  transperineal 
needle  insertion  and  designed  to  be  compatible  with  all  known  types  of  medical  imaging 
equipment.  This  includes  uncompromised  compatibility  with  MRI  scanners  of  the  highest 
field  strength,  size  accessibility  within  closed-bore  tunnel-shaped  scanners,  and  clinical 
intervention  safety. 

The  robot  is  designed  to  accommodate  various  end-effectors  for  different  percutaneous 
interventions  such  as  biopsy,  serum  injections,  or  brachytherapy.  The  first  end-effector 
developed  is  customized  for  fully-automated  low  dose  radiation  seed  brachytherapy. 

For  MRI  compatibility  the  robot  is  exclusively  constructed  of  nonmagnetic  and  dielectric 
materials  such  as  plastics,  ceramics,  and  rubbers  and  is  electricity  free.  The  system 
utilizes  a  new  type  of  motors  specifically  designed  for  this  application,  the  pneumatic 
step  motors  (PneuStep)  (25).  These  uniquely  provide  easily  controllable  precise  and 
safe  pneumatic  actuation.  Fiber  optic  encoding  is  used  for  feedback,  so  that  all  electric 
components  are  distally  located  outside  the  imager’s  room.  A  photograph  of  the  robot  on 
the  MR  scanner  table  is  shown  in  Figure  14. 


Figure  14:  MR  compatible  robot  developed  for  prostate  brachytherapy  at  Johns  Hopkins 

(simulation  of  a  clinical  procedure) 
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Motion  repeatability  tests  performed  in  the  MRI  scanner  show  mean  errors  of  0.076  mm. 
The  robot  was  found  to  be  compatible  with  several  different  imaging  devices  [25].  The 
clinical  utility  of  the  system  remains  to  be  investigated.  We  are  currently  evaluating 
needle  insertion  accuracy  with  in-vitro  and  ex-vivo  experiments.  An  animal  protocol  has 
already  been  filed  and  approved  for  in-vivo  studies.  An  institutional  review  board 
approval  was  also  received  for  human  trials  on  robot-scanner  ergonomics. 


3  Technical  Issues 

In  this  section  the  following  technical  issues  will  be  briefly  discussed: 
o  Imager  compatibility 
o  Registration 

o  Patient  movement  and  respiration 
o  Force  feedback 
o  Mode  of  control 

Imager  compatibility.  For  MRI  systems,  compatibility  can  be  achieved  by  using 
nonmagnetic  and  nonconductive  materials.  For  CT  systems,  radiolucency  of  the  end- 
effector  is  important  so  that  it  can  hold  the  instrument  on  the  scan  plane.  The  robot 
system  must  also  be  easily  interfaced  with  the  imaging  system  and  allow  quick  access  to 
the  patient  in  emergency  situations.  When  the  robot  system  is  actuated  it  should  not 
interfere  with  the  imaging  system.  The  kinematic  structure  of  the  robot  must  allow  it  to 
reach  inside  the  gantry,  which  is  one  reason  why  specially  designed  robots  are  needed 
for  these  procedures. 

Registration.  For  a  robot  to  target  the  anatomy  based  on  the  images,  the  coordinate 
system  of  the  robot  must  be  registered  to  the  coordinate  system  of  the  imaging  device.  If 
the  robot  is  permanently  attached  to  the  patient  table  of  the  imaging  device,  this 
registration  can  be  done  once  through  a  calibration  procedure.  If  the  robot  is  designed  to 
be  moved  from  one  imaging  device  to  another  or  to  be  placed  on  the  table  for  certain 
procedures,  fast  and  accurate  registration  techniques  are  required. 

Patient  movement  and  respiration.  A  limiting  problem  in  some  interventional 
techniques  is  organ  movement  due  to  respiration.  A  robotic  system  could  react  faster 
than  a  human  and  compensate  for  respiration.  This  compensation  would  first  require  that 
target  movement  due  to  respiration  be  recognized  and  accurately  tracked.  However,  for 
such  a  system  to  be  clinically  viable,  robustness  and  safety  would  need  to  be 
demonstrated. 

Force  feedback.  For  robotic  systems  that  include  an  active  needle  driver,  at  present 
there  is  no  force  feedback  provided  to  the  operator.  The  importance  of  this  feedback  is  a 
subject  of  current  debate  but  there  are  some  clinical  applications  where  it  seems 
desirable.  However,  there  is  no  accurate  way  to  measure  the  force  at  the  tip  of  surgical 
instruments  and  existing  force  feedback  devices  are  too  bulky  for  the  clinical 
environment.  In  addition,  friction  forces  on  the  cannula  and  tissue  during  insertion  are 
high  which  compromise  the  accuracy  of  force  feedback  measurements.  Therefore,  this 
topic  must  be  considered  a  research  issue  at  this  time. 

Mode  of  control.  The  “best”  user  interface  for  an  interventional  robot  has  yet  to  be 
determined.  For  many  procedures,  joystick  control  seems  well-suited  and  keeps  the 
physician  firmly  in  control.  Master/slave  systems  are  also  possible  and  as  noted  above 
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force  feedback  may  be  helpful  here.  However,  there  are  procedures  such  as  biopsy 
where  a  straight-line  trajectory  needs  to  be  followed  and  some  degree  of  autonomy 
seems  appropriate  if  robustness  can  be  achieved. 


4  Discussion  and  Summary 

As  percutaneous  procedures  with  cannulae  and  probes  under  image  guidance 
continue  to  increase  in  numbers  and  importance,  as  they  have  the  past  several  years, 
there  will  be  more  demand  for  technological  assistance.  In  this  role,  image-guided 
robots  may  have  a  place  and  this  place  needs  to  be  demonstrated  in  randomized  clinical 
trials.  Ten  years  ago  image-guided  procedures  largely  consisted  of  biopsies.  Over  the 
past  decade  interventional  techniques  have  blossomed  and  include  procedures  to  ablate 
tissue  with  energies  such  as  radiofrequency,  heat,  cold,  and  laser.  Reconstructive 
procedures  have  also  developed.  An  example  is  vertebroplasty  and  kyphoplasty  in 
which  methymethacrylate  is  injected  into  vertebra  to  increase  stability  and  to  reduce 
pain. 

Robots  have  some  potential  advantages  over  the  human  operator  in  certain  applications. 
Examples  include  working  in  hazardous  environments  such  as  imaging  rooms  where 
radiation  is  used.  During  fluoroscopic  or  CT  guided  procedures  the  operator  frequently 
advances  the  cannula  with  the  imaging  beam  off  and  then  acquires  an  image  to  identify 
the  current  position  of  the  tip.  Options  to  overcome  the  limitation  of  intermittent  imaging 
include  stand-off  devices  to  keep  the  operators  hands  out  of  the  direct  x-ray  beam. 

These  devices  are  clumsy  and  still  force  the  operator  to  be  too  close  to  the  radiation. 

During  percutaneous  radiotherapy  procedures  radioactive  seeds  or  probes  are  inserted 
into  the  patient.  These  are  dangerous  to  have  close  to  the  operator.  Other  potential 
uses  are  to  integrate  robots  with  image  guidance,  including  multimodality  integration, 
and  the  integration  of  tracking  technologies  such  as  optical  or  mechanical  trackers.  The 
robots  can  perform  active  guidance  in  procedures  where  path  planning  and  execution 
are  difficult  or  provide  a  zone  of  constraint  to  keep  the  operator  out  of  dangerous  areas. 
A  robot  can  also  be  integrated  with  active  control  to  compensate  for  motion  such  as 
respiration.  By  compensating  for  patient  motion  the  target  can  be  made  to  appear  static. 

To  be  accepted  in  clinical  practice,  however,  a  robot  must  be  intuitive  and  require 
minimal  operator  training.  It  must  also  be  quick  and  easy  to  setup  and  not  significantly 
increase  the  length  of  procedures.  Robot  must  also  be  cost  effective.  The  possibility  of 
performing  procedures  that  the  human  cannot  perform  and  are  clinically  necessary 
remains  an  ultimate  goal  for  medical  robotics.  Engineers  and  physicians  should  work 
together  to  create  and  validate  these  systems  for  the  benefits  of  patients  everywhere. 
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Abstract —  This  paper  presents  a  system-level  approach  to 
the  design  of  a  safety-critical  robotic  system  that  is 
sufficiently  safe  to  satisfy  human-subject  safety  criteria.  This 
system  design  approach  utilizes  preliminary  hazard  analysis, 
and  fault  tree  analysis,  and  was  successfully  applied  to  a 
dexterous  space  robot  designed  to  fly  on  NASA’s  space 
shuttle.  An  application  of  this  approach  to  a  shoulder 
rehabilitation  exoskeleton  will  be  presented  and  shown  to 
improve  the  safety  of  the  overall  system. 

I.  Introduction 

afety  has  always  been  a  paramount  concern  in  robotic 
systems,  particularly  when  there  is  a  potential  for 
humans  to  enter  the  robot’s  work  environment.  The  situation 
is  often  avoided  by  surrounding  the  workspace  of  the  robot 
with  a  detection  device  that  removes  power  to  the  robot  if 
someone  or  something  approaches.  Unfortunately,  this 
strategy  is  obsolete  when  it  comes  to  medical  applications 
such  as  surgery  and  rehabilitation  where  the  robot  must 
interact  with  the  patient  directly. 

In  recent  years,  robots  have  made  substantial  in-roads  in 
the  medical  field.  Devices  such  as  Da  Vinci  [1],  CyberKnife 
[2],  and  the  IM2  Robot  [3],  have  provided  researchers  and 
doctors  alike  with  capabilities  not  previously  available. 
These  additional  capabilities  have  also  brought  with  them 
the  issue  of  individual  patient  safety.  While  the  robot  must 
enforce  the  safety  of  the  patient  as  an  object  within  its 
workspace,  it  must  also  be  able  to  operate  upon,  or  assist  the 
patient,  contrary  to  most  robots  that  are  not  allowed  to  affect 
the  reachable  human  in  any  manner.  This  dichotomy  creates 
the  need  for  a  safety  system  that  can  allow  the  robot  to 
interact  with  the  patient,  but  also  enforce  all  necessary  safety 
precautions  at  the  same  time. 

In  perhaps  no  other  application  is  patient  safety  more 
acute  than  for  exoskeletons  in  which  the  human  is  basically 
encapsulated  in  a  robotic  device.  The  Maryland-Georgetown- 
Army  (MGA)  Exoskeleton,  shown  in  Figure  1,  is  an  arm 
exoskeleton  designed  to  treat  shoulder  pathology  in  a 
collaborative  project  between  Georgetown  University  and  the 
University  of  Maryland.  The  robot  has  five  degrees  of 
freedom,  powered  by  brushless  DC  motors  through  a 
harmonic  drive  train  capable  of  exerting  up  to  92  N-m  of 
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torque  at  the  shoulder.  Encoders  mounted  on  the  motors  and 
a  suite  of  force  sensors  at  the  shoulder,  elbow  and  wrist 
provide  input  to  the  control  system  to  realize  desired 
rehabilitation  protocols. 

This  basic  system  does  not  inherently  address  the  needs 
of  safety,  as  its  design  can  only  identify  certain  basic  robotic 
failures.  The  electromechanical  subsystems,  the  software 
subsystems  and  the  control  subsystem,  all  need  to  be 
examined  to  determine  overall  patient  safety.  This  paper  will 
detail  an  approach  to  generating  a  sufficiently  safe  system 
design  for  safety-critical  rehabilitation  applications.  The 
safety  system  of  the  MGA  Exoskeleton  will  be  used  as  an 
illustrative  example  of  this  approach. 

Previous  approaches  to  safety  system  design  will  first  be 
examined  in  Section  II,  and  an  existing  approach  based  on 
the  safety  system  for  a  dexterous  space  robot  will  be 
presented  in  Section  III.  An  overview  of  the  MGA  arm  and 
its  control  system  in  Section  IV  will  be  followed  by  an 
example  application  of  this  safety  system  approach  in 
Section  V.  This  section  will  also  examine  changes  in  the 
initial  system  design  necessary  to  enforce  safety. 


Figure  1  The  MGA  Exoskeleton  designed  for  shoulder  therapy. 


II.  Previous  Work 

Previous  medical  robotics  have  had  to  address  the  issue  of 
patient  safety  [4]  [5],  One  of  the  unique  aspects  of  the 
medical  robotic  system  presented  here,  is  that  the  human  in 
the  loop  is  the  patient.  With  surgical  or  radiological  systems 
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such  as  Da  Vinci  [1]  and  Cyberknife  [2],  a  patient  is  being 
“operated”  on  by  the  robot,  however,  a  clinician  is  directing 
the  robot.  With  the  MGA  system,  the  patient  is  both  the 
individual  upon  whom  the  robot  operates,  and  also  the 
individual  who  directs  the  robot. 

Unfortunately,  given  the  infancy  of  this  field,  there  is  no 
industry-standard  approach  to  designing  these  safety-critical 
robot  systems  [4]  [5].  Numerous  safety-critical  software 
systems  have  been  developed  and  deployed  in  other  domains 
ranging  from  aircraft  flight  management  systems  [6]  to 
nuclear  power  plants  [7].  Analytical  methods  similar  to  that 
presented  here  are  a  standard  and  accepted  practice  in  these 
domains,  when  identifying  and  characterizing  the  likelihood 
of  hazards  [8]. 

This  paper  presents  an  approach  that  was  successfully 
applied  to  a  space  robot  designed  to  fly  on  the  NASA  space 
shuttle  [9].  This  system  was  the  first  -  and  to  date  only  - 
American  robotic  system  to  be  certified  through  three  of  the 
four  phases  of  the  NASA  Space  Shuttle  Safety  Review 
process.  It  pioneered  a  solely  computer-based  hazard  control 
system  for  payloads  operating  on  the  shuttle. 

The  following  definitions  are  used  in  this  paper  [10].  A 
“failure”  is  an  abnormal  occurrence,  while  a  “fault”  is  a 
higher-order  event  caused  by  one  or  more  failures.  A 
“hazard”  is  a  system  state  and  other  environmental 
conditions  that  inevitably  leads  to  an  accident.  An  “accident” 
is  an  undesired  and  unplanned  event  that  results  in  a  level  of 
loss,  in  this  case,  injury  to  the  patient. 

III.  Approach 

The  process  used  to  determine  a  sufficiently  safe  system 
design  is  shown  in  Figure  2.  A  basic  system  design  that 
accomplishes  the  goals  of  the  project  is  first  examined  as 
part  of  a  Preliminary  Hazard  Analysis  (PHA)  [11].  A  Fault 
Tree  Analysis  (FTA)  [12]  is  then  developed  using  the 
system  design  and  the  list  of  hazards  generated  by  the  PHA. 
The  resulting  fault  trees  can  be  qualitatively  examined  to 
determine  if  the  system  is  “safe  enough”  for  the  project’s 
purposes.  If  not,  additional  components  are  typically  added 
to  the  system  in  an  effort  to  deal  with  the  specific  safety 
issues  raised  by  the  FTA  -  the  system  design  is  modified 
accordingly  -  and  the  process  starts  again.  Once  the  FTA 
results  show  that  the  project’s  safety  criteria  are  met,  the 
system  design  can  be  considered  complete. 

The  concept  of  “safe  enough”  is  one  that  the  specific 
project  must  establish.  It  is  not  possible  to  make  a  system 
absolutely  safe,  however,  if  the  likelihood  of  an  accident  is 
small  enough  or  the  consequences  of  an  accident  are 
negligible  enough,  the  system  may  be  considered  safe 
enough  [13]  [14].  At  some  point,  continuing  to  modify  a 
system  design  to  cope  with  ever  more  incredible  failures 
simply  results  in  an  excessively  complex  design,  and  a 
subsequent  reduction  in  overall  system  reliability  and/or 
safety.  In  order  to  develop  a  safe  system,  it  is  first 
important  to  understand  how  the  system  is  intended  to 
function. 


Figure  2  Approach  to  system  design  for  safety. 


IV.  Control  System 

The  MGA  Exoskeleton  has  two  operating  modes:  Virtual 
Reality  (VR)  Mode,  and  Physical  Therapy  (PT)  Mode.  In 
VR  Mode,  the  forces  exerted  at  the  hand  are  controlled  by 
interaction  with  a  virtual  environment  generated  by  a 
computer.  In  PT  Mode,  the  shoulder  is  exercised  about  an 
arbitrary  axis  through  the  glenohumeral  (GH)  joint  using  a 
preset  resistance  profile.  In  both  cases,  the  scapula  joint 
moves  independently  to  "  accommodate"  shoulder 
elevation/depression.  The  two  modes  generate  the  need  for 
contrasting  control  approaches  which  are  described  in  more 
detail  below  [15]. 

A.  Virtual  Reality  Mode 

Virtual  Reality  Mode  uses  computer-generated 
environments  to  simulate  daily  living  tasks  for  functional 
rehabilitation.  The  patient  views  the  simulated  task  and 
representation  of  their  arm  through  a  head  mounted  display 
while  the  exoskeleton  provides  haptic  feedback  to  the 
patient.  A  force  sensor  located  at  the  hand  gripper  senses  the 
forces  being  exerted  by  the  patient's  "contact"  with  the 
virtual  environment  and  relays  them  to  the  controller,  which 
moves  the  exoskeleton  in  response  to  the  interaction. 

The  admittance  controller  shown  in  Figure  3  is  used  to 
convert  sensed  contact  forces  into  motion  commands  [16] 
[17].  Signals  from  the  gripper  force-torque  sensor  and  elbow 
load  cells  are  input  to  the  virtual  environment,  which  then 
outputs  a  desired  velocity  for  the  wrist  and  angular  rate  of 
the  shoulder-elbow-wrist  (SEW)  plane  roll,  (j).  The  desired 
velocities  are  then  converted  into  desired  joint  velocities 
using  the  inverse  Jacobian,  which  are  then  tracked  using  a 
proportional-derivative  (PD)  control  law. 
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Figure  3  Admittance  controller. 


B.  Physical  Therapy  Mode 

Physical  Therapy  Mode  is  basically  a  programmable 
resistance  trainer  that  allows  the  patient  to  exercise  about  an 
arbitrary  shoulder  rotation  axis.  For  example,  to  treat  rotator 
cuff  injury,  therapists  often  prescribe  exercises  involving 
lateral/medial  rotation  of  the  shoulder.  Since  there  is  no 
single  joint  corresponding  to  shoulder  rotation,  the  exercise 
involves  all  three  axes  of  the  exoskeleton  shoulder.  Thus, 
the  controller  needs  to  affect  a  prescribed  resistance  profile 
about  the  desired  axis  while  preventing  rotations  about  the 
other  shoulder  axes. 

Since  there  is  no  direct  way  to  measure  the  torques  about 
the  shoulder  joints,  this  mode  uses  an  impedance  controller 
with  velocity  inputs  as  shown  in  Figure  4  [18]  [19].  In  this 
scheme,  the  joint  velocities  are  relayed  to  a  Jacobian,  Jqh,  to 
compute  the  Cartesian  velocities  about  the  GH  joint,  gh. 
These  velocities  are  then  multiplied  by  the  desired  resistance 
profile,  which  outputs  the  torques  required  about  the  GH 
axes,  Tgh.  These  torques  are  then  converted  into  the 
exoskeleton  joint  torques  using  the  Jacobian.  A  dynamic 
model  of  the  exoskeleton  runs  in  parallel  to  compute  gravity 
and  friction  feedforward  compensation  torques.  Since  only 
the  shoulder  axes  are  constrained,  the  elbow  pitch  is  left  free 
to  move  however  the  patient  desires. 


Figure  4  Impedance  controller. 


In  both  modes,  the  scapula  joint  is  controlled 
independently  from  the  arm  joints.  The  objective  is  to  keep 
the  exoskeleton  GH  joint  as  closely  aligned  with  the  human 
GH  joint  as  possible.  The  desired  scapula  angle  §  is 
determined  using  a  biomechanical  model  of  the  scapula 
motion  based  on  the  motion  of  the  GH  joints.  A  tracking 
controller  then  drives  the  scapula  joint  to  the  desired  angle. 
While  only  approximate  tracking  of  the  GH  joint  is  possible 
using  a  single  rotary  joint,  this  scenario  is  preferable  to  the 
uncompensated  motion  observed  in  simulations  of  other 
powered  orthoses  such  as  MULOS  [20]. 

The  control  system  defines  the  minimum  suite  of  sensors 
and  actuators  that  are  required  to  carry  out  operations,  a 
partial  diagram  of  which  is  shown  in  Figure  5.  While  this 
initial  system  design  provides  for  operational  capability,  it 
must  be  further  refined  to  ensure  the  patient’s  safety.  The 
next  section  will  detail  an  approach  to  refining  a  basic 
system  design  in  order  to  ensure  safety  of  the  patient. 


Control  station 


Figure  5  Initial  system  design.  This  is  the  minimum  suite  of  sensors  and 
actuators  required  for  operations.  For  clarity,  only  major  system 
components  are  shown. 


V.  System  Safety 

A  preliminary  hazard  analysis  of  the  MGA  system 
identified  three  potential  hazards  (in  this  context,  "excessive” 
means  an  unhealthy  level,  leading  to  injury): 

•  Moving  the  patient  outside  their  safe  position  range. 

•  Moving  the  patient  at  an  excessive  velocity. 

•  Applying  excessive  torque  to  the  patient  or,  conversely, 
allowing  the  patient  to  apply  excessive  torque  against 
the  robot. 

The  project  safety  criteria  specify  that  no  single  failure  can 
cause  a  hazard,  and  that  the  system  is  “fail-safe”.  A  fail-safe 
system  is  one  that  will  achieve  a  safe  state  in  the  presence  of 
a  detected  fault  [9].  When  a  fault  is  detected,  the  system  will 


254 

Page  65 


IEEE  Inter  Conf  on  Rehabilitation  Robotics 


either  a)  halt  arm  motion  and  hold  the  current  position,  or  b) 
safe  the  arm  by  removing  power  to  the  motors.  Removing 
power  has  a  more  pronounced  effect  on  the  patient,  as  they 
now  have  to  hold  up  the  weight  of  the  device.  Thus,  this 
approach  is  used  only  when  more  severe  failures  occur  or 
when  a  reliable  arm  halt  cannot  be  guaranteed.  The  state  of 
the  system,  both  patient  and  robotic,  is  safe  for  the  patient  if 
either  a)  or  b)  occurs. 

The  hazards  identified  by  the  PHA  each  constitute  “top 
events”  from  which  FTA  can  begin.  Each  top  event  is 
considered  individually,  and  the  immediate,  necessary,  and 
sufficient  causes  by  which  this  event  could  occur  are 
identified.  These  immediate  events  will  summarily  be 
examined  for  their  causing  events,  and  this  step  by  step 
analysis  continues  until  individual  component  failures  are 
reached.  These  component  failures  are  the  basic  causes  that, 
when  combined  in  the  manner  indicated  by  the  fault  tree, 
guarantee  that  the  top  level  hazard  will  occur.  The  symbols 
used  in  this  work  to  represent  fault  tree  events  and  gates  are 
shown  in  Figure  6.  Further  details  of  fault  trees  and  their 
construction  can  be  found  in  [12]. 


BASIC  EVENT.  A  basic  initiating  fault 
requiring  no  further  development. 


INTERMEDIATE  EVENT.  A  fault  event  that  occurs  due  to 
one  or  more  antecedent  causes  acting  through  logic  gates. 


□ 


AND  gate.  Output  fault  occurs 
if  all  input  faults  occur. 


OR  gate.  Output  fault  occurs  if 
at  least  one  input  fault  occurs. 


Figure  6  Symbols  used  in  fault  trees. 


A.  Moving  the  patient  outside  their  safe  position  range 

A  fault  tree  developed  from  the  initial  system  design  of 
Figure  5,  and  the  top  event  “Moving  the  patient  outside 
their  safe  position  range”,  is  shown  in  Figure  7.  This  partial 
fault  tree  provides  an  example  in  which  a  single  fault  could 
cause  this  hazard.  The  top  event  can  be  caused  by  any  one 
of  numerous  possible  intermediate  events,  due  to  the  OR 
gate  attached  to  the  top  event.  The  intermediate  event 
shown,  “Uncommanded  motion  due  to  joint  runaway”,  can 
be  caused  solely  by  a  failure  of  the  incremental  encoder, 
which  is  a  primary  component  of  the  control  law  used  to 
drive  the  motor. 

This  scenario  fails  the  project  safety  criteria,  and  so 
additional  components  were  added  to  the  system  and  the 
PHA  and  FTA  were  repeated.  The  modified  system  design  is 
shown  in  Figure  8,  where  the  shaded  components,  an 
absolute  encoder  and  a  power  amplifier,  are  additions  over 
the  initial  system  design.  Note  that  for  clarity,  additional 
safety  components  such  as  emergency  stop  measures  are  not 
shown. 


Figure  7  Fault  tree  for  the  initial  system  design  and  the  top  event 
“Moving  the  patient  outside  their  safe  position  range”.  This  fault  tree 
shows  that  a  single  fault,  that  of  the  incremental  encoder,  could  cause 
the  top  event  to  occur. 

The  fault  tree  for  this  top  event  and  the  modified  system 
design  is  shown  in  Figure  9.  This  fault  tree  considers  the 
addition  of  a  second  encoder  and  a  software-based  divergence 
check  to  the  system  design.  The  divergence  check  is 
designed  to  detect  a  failed  encoder  by  comparing  the  values 
of  the  two  encoders,  and  flagging  a  fault  if  they  differ  by 
more  than  a  prescribed  tolerance.  This  fault  tree  demonstrates 
that  the  addition  of  the  second  encoder  and  the  encoder 
divergence  check  will  satisfy  the  project  safety  criteria  for 
this  hazard:  no  one  fault  is  capable  of  producing  the  hazard. 


Cunt  txil  stilt  in  it 


Figure  8  Modified  system  design  with  additional  components  to  satisfy 
project  safety  criteria.  The  additions  over  the  initial  system  design  of 
Figure  5  are  shaded.  For  clarity,  only  major  system  components  of 
relevance  are  shown. 

While  the  modified  system  design  does  prevent  a  single 
failure  from  causing  this  hazard,  closer  examination  of 
Figure  9  shows  that  a  double  failure  could  still  cause  the 
hazard.  If  both  encoders  fail  in  such  a  way  that  they  output 
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almost  the  identical  same  value  they  would  pass  the  encoder 
divergence  check.  While  this  failure  combination  is  possible, 
particularly  for  certain  values  (depending  on  the  encoders 
construction,  0  or  -1  are  likely  candidates),  it  is  highly 
unlikely  to  occur  at  the  same  time,  and  thus  could  be 
deemed  an  “incredible”  failure  and  removed  from  further 
analysis.  While  further  modifications  to  the  system  design, 
such  as  a  third  encoder,  may  enable  detection  of  such 
situations,  the  additional  system  complexity  may  be 
unwarranted  as  well  as  potentially  contributing  to  lower 
system  reliability.  The  tradeoff  between  these  measures  is 
beyond  the  scope  of  this  paper. 


Figure  9  Fault  tree  for  the  modified  system  design  and  the  top  event 
“Moving  the  patient  outside  their  safe  position  range”.  This  fault  tree 
indicates  that  two  simultaneous  faults  are  required  for  the 
intermediate  event  shown  to  cause  the  top  event  to  occur. 


To  help  determine  the  overall  likelihood  of  such 
incredible  failures  occurring,  the  fault  trees  may  be 
quantitatively  evaluated.  It  must  be  noted,  however,  that 
FTA  is  more  a  qualitative  technique,  and  that  “its  absolute 
accuracy  is  generally  secondary  to  identification  of  failure 
sequences”  [21].  Quantitative  analysis  may  therefore  be 
beneficial  in  simply  ranking  failures  by  probabilistic 
likelihood,  versus  using  the  output  probabilities  as  absolute 
indications  of  safety  [10]. 

B.  Moving  the  patient  at  an  excessive  velocity 
The  fault  trees  for  this  hazard  are  very  similar  in  structure 
to  those  for  the  previous  hazard.  This  is  primarily  due  to  the 
system  computing  velocity  based  on  sequential  encoder 
readings,  and  hence  there  are  identical  measures  to  sense 
excessive  velocity  or  to  detect  a  failed  component  that 
contributes  to  velocity  sensing.  Thus,  this  hazard  is  not 
considered  further  here. 


C.  Applying  excessive  torque  to  the  patient 
A  fault  tree  for  the  initial  system  design  and  the  top 
event,  “Applying  excessive  torque  to  the  patient”,  is  shown 
in  Figure  10.  A  single  fault  of  the  servo  controller,  which  is 
responsible  for  providing  power  to  the  motor,  is  capable  of 
producing  uncommanded  motion  and  hence,  potentially, 
applying  excessive  torque  to  the  patient.  The  fault  tree  of 
Figure  1 1  is  for  the  modified  system  design,  and  shows  the 
addition  of  a  separate  power  amplifier  with  built-in  motor 
current  sensor,  and  a  software-based  motor  power  check. 
This  power  check  compares  the  motor  current  draw  with  the 
requested  output  of  the  servo  controller,  to  determine  if 
either  component  is  at  fault.  This  fault  tree  indicates  that  the 
project  safety  criteria  are  satisfied  by  these  additions. 


Figure  10  Fault  tree  for  the  initial  system  design  and  the  top  event 
“Applying  excessive  torque  to  the  patient”.  This  fault  tree  shows  that  a 
single  fault,  that  of  the  servo  controller,  could  cause  the  top  event  to 
occur. 


Figure  11  Fault  tree  for  the  modified  system  design  and  the  top  event 
“Applying  excessive  torque  to  the  patient”.  This  fault  tree  indicates 
that  two  simultaneous  faults  are  required  for  the  intermediate  event 
shown  to  cause  the  top  event  to  occur. 
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VI.  Conclusion 

The  methodology  presented  here  allows  system  designers 
to  produce  an  overall  system  design  that  is  sufficiently  safe 
to  satisfy  the  project’s  safety  criteria.  This  approach  will 
often  result  in  additional  components  being  added  to  a 
system,  to  ensure  that  the  safety  system  can  detect  failures 
and  act  accordingly. 

The  safety  system  for  the  MGA  Exoskeleton  consists  of  a 
suite  of  interwoven  hardware  constraints  and  devices  (e.g. 
secondary  encoders,  slip  clutch),  electronic  checks  (e.g. 
encoder  illegal  states),  and  software  checks  (e.g.  encoder 
divergence  checks,  electronic  component  heartbeats).  The 
system  design  to  date  satisfies  the  project  safety  criteria,  and 
indeed,  carries  over  much  of  the  safety  approach  and  design 
elements  from  its  space  flight  predecessor. 
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ABSTRACT 

Workflow  analysis  has  the  potential  to  dramatically  improve  the  efficiency  and  clinical  outcomes  of  medical  procedures. 
In  this  study,  we  recorded  the  workflow  for  nerve  block  and  facet  block  procedures  in  the  interventional  radiology  suite 
at  Georgetown  University  Hospital  in  Washington,  DC,  USA.  We  employed  a  custom  client/server  software  architecture 
developed  by  the  Innovation  Center  for  Computer  Assisted  Surgery  (ICCAS)  at  the  University  of  Leipzig,  Germany. 
This  software  runs  in  an  internet  browser,  and  allows  the  user  to  record  the  actions  taken  by  the  physician  during  a 
procedure.  The  data  recorded  during  the  procedure  is  stored  as  an  XML  document,  which  can  then  be  further  processed. 
We  have  successfully  gathered  data  on  a  number  if  cases  using  a  tablet  PC,  and  these  preliminary  results  show  the 
feasibility  of  using  this  software  in  an  interventional  radiology  setting.  We  are  currently  accruing  additional  cases  and 
when  more  data  has  been  collected  we  will  analyze  the  workflow  of  these  procedures  to  look  for  inefficiencies  and 
potential  improvements. 

Keywords:  Surgical  Workflow,  Workflow  Management  Tools,  Workflow  Optimization,  Surgical  PACS 

1.  INTRODUCTION 

The  ability  to  systems  engineer  clinical  environments  in  terms  of  formal  workflow  descriptions  of  processes  and 
procedures  has  significant  implications.  Workflow  analysis  has  the  potential  to  dramatically  improve  the  efficiency  and 
clinical  outcomes  of  medical  procedures.  From  an  operations  standpoint,  it  allows  inefficiencies  to  be  identified  and 
remedied  more  quickly.  From  a  technology  perspective,  it  enables  a  methodical  and  scientific  approach  to  the 
specification,  simulation,  design  and  prototyping  of  new  technology,  allowing  development  to  occur  more  efficiently.1,2 
In  either  case,  it  allows  the  impact  of  a  new  strategy  to  be  assessed  objectively. 

The  application  of  workflow  analysis  to  the  clinical  environment  is  only  beginning  to  emerge  and  is  likely  to  gain  more 
interest  in  the  upcoming  years.  Significant  workflow  analysis  research  is  currently  being  done  by  the  Innovation  Center 
for  Computer  Assisted  Surgery  (ICCAS)  in  the  area  of  surgical  interventions  as  the  basis  for  the  development  of 
computer  assisted  surgery  (CAS)  systems.1,2,3  ICCAS  has  focused  its  efforts  on  modeling  surgical  workflows  by 
recording  data  from  real  world  surgical  interventions  that  can  be  associated  with  well-defined  task  of  the  intervention 
with  high  granularity.  The  development  of  a  computerized  workflow  editor  tool  to  record  workflow  data  is  crucial  to  the 
capability  of  the  recorder  to  capture  fine-grained  data.  The  tool  draws  upon  ontologies  developed  for  different  surgical 
disciplines  and  also  includes  a  visualization  tool  to  analyze  the  data. 

Interventional  Radiology  (IR)  can  benefit  from  a  similar  effort.  IR  is  an  image-guided  therapy,  and  can  take  advantage  of 
any  and  all  imaging  modalities,  and  accompanying  computer  and  mechanical  enhancements.  The  capability  to  model  IR 
procedures  will  allow  new  technologies  to  be  developed  and  evaluated  more  quickly.  Additionally,  the  boundaries 
between  IR  and  surgery  are  blurring.  While  IR  procedures  include  increasingly  more  invasive  therapeutics,  such  as 
tumor  ablation,  surgeons  are  adopting  the  techniques  and  principles  of  IR,  and  are  relying  more  heavily  on  intra¬ 
operative  imaging.  Thus,  it  is  timely  to  apply  the  workflow  analysis  approach  developed  for  surgical  interventions  to  IR 
procedures. 
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In  this  study,  the  ICCAS  workflow  tool  is  adapted  to  the  specific  requirements  of  IR  workflow.  It  is  then  used  to  record 
workflow  data  for  nerve  block  and  facet  block  procedures  in  the  IR  suite  at  Georgetown  University  Hospital  in 
Washington,  DC.  The  intent  of  the  study  is  to  determine  the  feasibility  of  recording  workflows  in  an  IR  setting  and  to 
analyze  the  workflow  of  these  procedures,  identify  inefficiencies,  and  potentially  improve  the  efficiency  of  these 
procedures.  This  paper  describes  the  development,  of  an  IR-specific  workflow  tool  and  the  results  of  using  it  to  record 
workflow  in  an  IR  setting. 


2.  METHODS 

We  recorded  workflows  using  a  software  package  developed  at  ICCAS  that  facilitates  the  structured  recording  of 
surgical  workflows.  The  ICCAS  software  uses  a  workflow  editor  to  support  the  difficult  process  of  dealing  with  the 
complex  relationships  and  concurrencies  that  occur  during  surgical  interventions.  It  then  generates  a  structured 
description  of  the  intervention  in  XML  format  that  can  be  used  for  visualization  and  further  analysis.  Although  the 
software  was  developed  to  record  surgical  workflow,  its  design  makes  it  easy  to  adapt  to  other  disciplines  by  changing 
the  underlying  ontology.2  Since  IR  workflow  had  not  been  previously  recorded,  the  approach  was  to  first  start  with  a 
simple  procedure  that  is  performed  with  some  frequency.  Nerve  and  facet  block  procedures  were  selected  since  they  fit 
the  outlined  criteria.  In  this  way,  the  ontology  could  be  developed  and  integrated  with  the  workflow  application.  Actual 
recording  could  then  be  conducted  with  enough  frequency  to  provide  an  understanding  of  the  feasibility  of  recording 
workflow  data  in  the  IR  setting. 

2.1  Developing  the  interventional  radiology  (IR)  ontology 

Although  commonalities  exist  between  IR  procedures  and  surgical  interventions,  there  are  significant  enough  differences 
that  the  ICCAS  tool  must  be  modified  to  support  IR.  Therefore,  the  first  step  was  to  generate  the  appropriate  ontology 
for  IR  workflow  of  nerve  block  and  facet  block  procedures.  After  a  thorough  analysis  through  observation  of  multiple 
nerve  and  facet  blocks  procedures,  the  procedures  were  broken  down  into  specific  tasks  each  comprised  of  four 
components  (1)  participants  involved  (2)  actions  performed  (3)  instruments  used  and  (4)  anatomic  structures  treated.  By 
combining  the  components  across  all  identified  tasks  in  a  block  procedure,  comprehensive  lists  for  each  component  were 
generated  (See  Figure  1).  Each  task  executed  during  a  procedure  is  defined  as  a  combination  of  items  from  each  of  these 


Participants  Involved 

Anatomic  Structures  Treated 

-  operator  (physician) 

Nerve  Blocks 

-  C1,C2,C3,C4,C5,C6,C7,C8 

Actions  Performed 

-  T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T1 1,T12 

-  acquire  image 

-  L1,L2,L3,L4,L5 

-  place 

Facet  Blocks 

-  insert 

-C 1  -2,C3-4,C5-6,C7-8 

-  remove 

-T1-2,T3-4,T5-6,T7-8,T9-10,T11-12,T12-L1 

-  insert  syringe 

-L1-2,L3-4,L5-S1 

-  adjust  angle  and  advance 

-  inject 

Side 

-  mix  drugs 

-right 

-left 

-right/left 

Instruments  Used 

-  local  anesthesia  needle 

-  syringe 

Spine  Levels 

-  local  anesthesia  needle  with 

OCervical 

syringe 

T=Thoracic 

-  block  needle 

L= Lumbar 

-  fluoroscopy 

S=Sacral 

Figure  1  -  Specific  ontology  developed  for  IR 
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lists.  Once  the  lists  were  completed,  ICC  AS  incorporated  the  IR  ontology  into  the  workflow  editor.  The  resulting 
interface  is  shown  in  Figure  2. 
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Figure  2  -  ICC  AS  workflow  editor  modified  to  support  IR  nerve  and  facet  block  procedures 


2.2  Technology  description 

For  this  study,  we  chose  to  run  the  software  on  a  Tablet  PC,  which  is  portable  and  has  a  touch  screen  display  for  easy 
data  entry.  The  workflow  editor  is  run  as  a  web  application.  It  is  programmed  in  Hypertext  Preprocessor  (PHP)  which 
supports  the  development  of  web-based  software  applications.  We  used  XAMPP,  an  easily  installable  and  configurable 
Apache  (A)  distribution  with  combined  MySQL  (M),  Perl  (P)  and  PHP  (P)  support.  The  X  stands  for  different 
distributions  for  Linux,  MacOS  and  Windows.4  We  installed  it  on  the  PC  in  order  to  simplify  the  architecture  and 
hardware  requirements.  For  recording  multiple  workflows  on  multiple  clients,  a  dedicated  server  and  wireless  network 
could  be  utilized.  Pointing  the  Mozilla  Firefox  web  browser  to  the  PHP  pages  launches  the  application. 

2.3  Workflow  recording 

Before  recording  can  begin,  the  user  must  first  collect  data  about  the  procedure.  The  workflow  editor  prompts  the  user  to 
enter  information  about  the  recording  person  (user),  location  of  the  procedure,  discipline,  diagnosis,  therapy,  participants, 
and  patient.  Once  this  information  has  been  entered,  recording  is  initiated  by  selecting  the  ‘Start’  button  to  signify  the 
start  of  the  procedure.  During  the  block  procedure,  the  user  selects  the  ‘Start  New  Activity’  button  to  indicate  when  a 
new  task  is  beginning.  The  software  records  this  timestamp  as  the  start  time  of  the  task.  The  user  then  uses  the  graphical 
user  interface  (GUI)  to  select  the  items  from  each  list  that  define  the  task,  namely  the  participant  (e.g.  surgeon),  action 
performed,  the  instrument  used,  and  the  anatomic  structure  involved.  Once  the  participant  has  completed  the  action,  the 
user  selects  the  ‘Stop’  button  and  the  software  records  this  time  as  the  stop  time.  This  sequence  is  repeated  for  each  task 
of  the  block  procedure.  Some  procedures  may  involve  multiple  tasks  occurring  concurrently.  The  workflow  editor 
handles  this  situation  by  providing  a  timeline  of  tasks  on  the  screen.  If  a  new  task  starts  before  the  previous  task  ends, 
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the  user  simply  selects  the  ‘Start  New  Activity’  button  without  stopping  the  previous  task.  The  user  may  then  use  the 
timeline  to  access  and  signal  completion  of  previous  tasks. 

When  the  block  procedure  ends  and  recording  is  completed,  the  user  selects  the  ‘Save’  button.  The  workflow  editor 
creates  an  XML  file  consisting  of  a  header  and  a  body  that  contains  all  of  the  recorded  data.  The  header  contains  the 
contextual  data  about  the  procedure  that  was  entered  before  the  start  of  the  procedure.1  It  includes  the  following  data 
elements: 

(a)  the  discipline  with  the  child  elements 

(al)  diagnosis 
(a2) therapy 

(a3) participant  with  the  elements  position  (e.g.  interventionalist,  technologist,  nurse)  and  a  note  field 
(a4)  a  note  field  for  discipline  related  information 

(b)  the  date  of  recording , 

(c)  the  place  of  recording ,  with  the  child  elements 

(cl)  country 

(c2)  city 

(c3)  hospital 

(c4)  operating  theatre 

(c5)  a  note  field  for  recording  place  related  information 

(d)  the  recording  person  with  the  child  elements 

(dl) first  name 
(d2)  last  name 

(d3)  status  (e.g.  medical  student,  recording  experience) 

(d4)  a  note  field  for  recording  person  related  information 

(e)  an  input  field  for  notes  regarding  the  whole  intervention. 

In  the  body  of  the  XML  file,  the  data  are  partitioned  into  tasks  that  represent  the  work  steps  of  the  IR  procedure.1  Each 
task  has  the  following  structure: 

(a)  the  tasktime  with  the  child  elements 

•  start  time 

•  stop  time 

•  duration 

(b)  an  actuator  that 

•  has  the  same  position  as  indicated  by  the  participant  element  inside  discipline  (e.g.  interventionalist, 
technologist,  nurse)  and  a  note  field 

•  a  note  field  for  participant  related  information 

•  an  element  that  indicates  various  used  body  parts  (in  regarding  of  granularity  level  (ii)  or  (iii)),  such  as 
‘left  hand’,  ‘right  hand’.  The  consideration  of  other  parts  of  the  body  like  ‘right  foot’  for  the  operation 
of  foot  pedals  or  ‘gaze’  for  the  gathering  of  information  from  monitors  is  also  possible. 

(c)  the  accomplished  activity , 

(d)  the  instrument  used  in  the  work  step, 

(e)  the  treated  anatomic  structure , 

(f)  an  input  field  for  notes 


Figure  3a  and  3b  illustrate  the  XML  file  The  XML  data  can  also  be  transformed  into  a  two-dimensional  Scalable  Vector 
Graphic  (SVG)  for  semantic  analysis  of  the  single  workflow  activities.3  It  allows  the  workflow  to  be  generalized  and 
analyzed,  allowing  suggestions  to  be  made  to  improve  the  efficiency  of  the  procedure. 
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<rec  workflow  workflowID- 'wfll32676025687.2"> 

<discipline  discipline- 'Interventional  Radiology"> 

<diagnosis>pain</diagnosis> 

<therapy>nerve  block</therapy> 

<task  taskID- T"> 

<participant> 

<tasktime> 

<position>operator</position> 

<starttime>l  122395258.1 1 1 7</starttime> 

<name>Dr.  E.  Nostaw</name> 

<stoptime>l  122395261 .2262</  stoptime> 

<note></note> 

<duration>3.1144998073578</duration> 

</participant> 

</tasktime> 

<patient> 

<actuator> 

<age>84</age> 

<position>operator</position> 

<sex>m</sex> 

<usedbodypart>operator</usedbodypart> 

<position>prone</position> 

</actuator> 

<note></note> 

<activity> 

</patient> 

<action>place</  action> 

<note></note> 

</activity> 

</discipline> 

<instrument> 

<rec_date>2005-ll-22</rec_date> 

<usedinstrument>pointer</usedinstrument> 

<rec_location> 

</instrument> 

<country>U  SA</country> 

<anatomic_structure> 

<city>Washington,  DC</city> 

<treatedStructure> 

<institution>Georgetown  Univ  Hospital</institution> 

<structure>N  erve</  structure> 

<building>C  C  C</building> 

<level>L3</level> 

<operatingtheatre>IR  Room  6</operatingtheatre> 

<side>Left</  side> 

<note></note> 

</treatedStructure> 

</rec_location> 

</ anatomic_structure> 

<rec_by> 

</task> 

<name>Donald  Siddoway</name> 

<status>medical  student</status> 

<note></note> 

</rec_by> 

Figure  3  a  -Header  component  of 
XML  file  for  IR  workflow 


Figure  3a  -Body  component  of 
XML  file  for  IR  workflow 


3.  RESULTS 


3.1  Workflow  results 

IR  workflow  was  recorded  for  7  nerve  block  procedures  and  1  facet  block  procedure  (Table  1).  Analysis  indicates  that 
the  tasks  are  short  in  duration  and  that  the  idle  time  between  tasks  is  very  short.  The  workflow  editor  successfully 
generated  the  appropriate  XML  files.  These  data  files  provide  a  thorough  record  of  all  the  actions  taken  by  the 
interventional  radiologist  during  block  procedures.  Figure  4  shows  a  sample  of  an  XML  workflow  file  with  the 
accompanying  SVG  representation.  These  can  then  be  analyzed  in  order  to  assess  the  efficiency  of  the  procedure,  both 
on  the  basis  of  the  individual  case,  and  for  the  procedure  in  general. 

3.2  Usability  of  the  workflow  editor  in  IR 

The  tablet  PC  is  a  good  choice  for  recording  workflow  because  it  offers  access  to  an  external  keyboard  and  a  touch 
screen.  Having  access  to  a  keyboard  makes  it  easy  to  enter  text  data  about  the  procedure  while  the  tablet  mode  is  better 
suited  for  the  workflow  recording  -  it  is  easier  to  hold  in  the  procedure  room,  and  the  touch  screen  display  eliminates  the 
need  for  a  mouse  and  keyboard.  With  respect  to  recording  the  workflow  of  block  procedures,  we  found  that  the  high 
level  of  granularity  defined  for  the  workflow  editor  combined  with  the  very  rapid  pace  of  the  block  procedures  made 
workflows  difficult  to  record.  In  general,  the  tasks  were  very  short  in  duration  and  the  idle  time  between  tasks  was  even 
shorter  making  it  difficult  for  the  recording  person  to  accurately  capture  the  end  of  one  task  and  the  start  of  a  new  one.  It 
became  particularly  difficult  when  several  short-duration  tasks  occurred  sequentially.  For  example,  when  positioning  the 
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needle,  the  steps  of  adjusting  the  needle  and  taking  an  X-ray  image  alternate  until  the  needle  is  properly  positioned. 
Neither  of  these  steps  takes  very  long  and  the  idle  time  between  them  is  short.  It  was  not  always  possible  to  capture  the 
task  accurately  or  at  all,  thereby,  decreasing  the  reliability  of  the  data. 


Procedure 

Type 

Number  of 
tasks/procedure 

Total 

time/procedure 

(seconds) 

Mean  duration 
time/task 
(seconds) 

Mean  idle 
time/task 
(seconds) 

Workflow  1 

nerve 

31 

342.0 

7.7 

3.3 

Workflow2 

nerve 

31 

414.0 

9.1 

3.2 

Workflow3 

nerve 

21 

240.0 

3.8 

3.1 

Workflow4 

facet 

10 

192.0 

15.1 

4.0 

Workflow5 

nerve 

25 

453.0 

13.4 

3.5 

Workflow6 

nerve 

24 

257.0 

7.9 

2.8 

Workflow7 

nerve 

10 

245.0 

17.8 

6.6 

Workflow8 

nerve 

21 

265.0 

8.1 

4.9 

Mean 

21.6 

301.0 

10.4 

3.9 

Table  1  -  IR  workflow  for  eight  procedures 


The  GUI  itself  exacerbated  the  issue  because  for  each  new  task,  the  recording  person  must  select  a  minimum  of  four 
buttons.  Sometimes  the  task  would  be  completed  before  all  selections  could  be  completed.  There  were  also  some 
problems  adapting  the  software  to  our  particular  recording  devices.  Most  notably,  the  GUI  did  not  fit  on  the  screen,  so 
some  scrolling  was  required  to  press  some  of  the  buttons.  This  further  slowed  the  recording  process.  Some 
modifications  were  made  to  the  GUI  to  allow  for  quicker  recording.  One  such  optimization  involved  the  selection  of  the 
involved  anatomic  structure.  For  block  procedures,  the  anatomic  structure  changes  only  once  or  not  at  all.  To  address 
this  issue,  the  software  was  changed  so  that  when  an  anatomic  structure  was  selected,  it  would  remain  selected 
throughout  the  procedure. 


4.  DISCUSSION 

As  a  proof  of  concept,  this  study  has  been  successful  in  demonstrating  the  feasibility  of  recording  workflow  in  IR  and  in 
providing  important  lessons  learned  that  will  be  critical  to  improving  subsequent  workflow  studies.  This  block  procedure 
workflow  study  has  established  that  the  process  of  recording  workflow  in  IR  is  feasible.  The  ICC  AS  workflow  editor 
was  able  to  be  adapted  to  IR  workflow,  resulting  in  the  appropriate  XML  data  files  and  accompanying  SVG  graphical 
representations.  Recording  was  not  disruptive  to  the  any  aspect  of  the  block  procedure  itself  and  from  an  ergonomic 
perspective  the  tablet  PC  was  a  good  match  for  recording  workflow. 

The  primary  lesson  learned  was  that  the  physical  act  of  recording  must  be  able  to  support  the  selected  granularity  of  the 
procedure.  It  is  critical  to  balance  the  type  of  procedure,  the  granularity  of  described  tasks  within  the  procedure  and  the 
interface  to  the  workflow  editor.  The  block  procedures  were  selected  because  they  represent  straightforward  procedures 
with  very  little  concurrent  tasks  and  they  are  performed  weekly  at  Georgetown  University  Hospital.  The  basis  for  this 
choice  was  that  it  would  allow  rapid  development  of  the  necessary  ontology  and  that  recording  could  be  conducted 
regularly,  providing  data  for  workflow  analysis.  The  problem  encountered  is  that  in  setting  up  the  ontology,  we  chose  a 
description  of  high  granularity.  We  did  not  recognize  that  the  rapid  pace  of  the  procedure  coupled  with  such  a  detailed 
description  would  cause  difficulty  in  accurately  recording  the  procedure.  Future  studies  must  take  this  into  account 
whether  it  means  modifications  to  the  recording  interface  or  changing  the  granularity  of  the  procedure  description 

For  this  study  specifically,  some  of  the  issues  can  be  remedied.  The  ontology  can  be  collapsed  where  commonly  used 
steps  occur  in  a  specific  sequence  extending  the  duration  time.  It  is  also  possible  to  combine  several  commonly  used 
steps  in  the  ontology  into  fewer  button  presses.  It  may  also  be  possible  to  integrate  voice  commands,  rather  than  button 
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pushes,  to  allow  for  quicker  recording  and  to  allow  the  person  using  the  workflow  editor  to  always  watch  the  procedure. 
Video  recordings  could  also  be  used  to  validate  workflow  data  and  make  corrections  where  necessary. 


<task  taskID="19"> 

<tasktime> 

<starttime>l  122394250.5829</  starttime> 
<stoptime>1122394255.7404</stoptime> 
<duration>5.1575002670288</duration> 
</tasktime> 

<actuator> 

<position>operator</position> 

<usedbodypart>operator</usedbodypart> 

</actuator> 

<activity> 

<action>acquire  image</action> 
</activity> 

<instrument> 

<usedinstrument>fluoro</usedinstrument> 

</instrument> 

<anatomic_stracture> 

<treatedStructure> 

<structure>N  erve</  structure> 
<level>L4</level> 

<side>Left</  side> 

</treatedStructure> 

</  anatomic_stracture> 

</task> 

<task  taskID='f20n> 

<tasktime> 

<starttime>l  122394256.7 118</starttime> 
<stoptime>l  122394261. 9593</stoptime> 
<duration>5.2474999427795</duration> 
</tasktime> 


00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 

00:04 


7 acquire  linage  ^ 

f luoro 

Necve  l4  Left 


Gidjast  anqle  and  advance^ 
block  flee die 
werve  L4  Left 


7 acquire  intake  ^ 

f luoro 

Nerve  l4  Left 


Figure  4  -  Text  on  the  left  represents  recorded  data  for  tasks  19-20  in  a  typical  case  with  the  corresponding  SVG 
representation  on  the  right.  Each  blue  blocks  represents  a  task  with  the  top  of  block  indicating  the  start  time  and  the 
bottom  of  the  block  indicating  the  stop  time  (the  height  of  the  block  is  the  duration).  The  time  between  blocks 
represents  idle  time.  The  SVG  representation  allows  one  to  quickly  visualize  the  workflow. 


5.  CONCLUSION 

Workflow  recording  and  analysis  in  IR  is  a  work-in-progress.  It  has  the  potential  to  be  a  valuable  tool  in  creating  formal 
descriptions  of  IR  procedures  that  can  then  be  used  to  evaluate  new  diagnostic  and  therapeutic  strategies,  and  launch  the 
development  of  new  devices  or  image-guided  systems  (e.g.  using  a  telemanipulator  compared  to  manual  intervention). 
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Workflow  is  also  at  the  foundation  of  integrating  IR  with  other  clinical  services  using  the  Integrated  Health  Enterprise 
(IHE)  approach  and  standards  development  such  as  Digital  Imaging  and  Communications  in  Medicine  (DICOM).5  6  The 
intent  is  to  refine  the  work  done  on  block  procedures  by  modifying  granularity  of  the  procedure  description  and  further 
improve  the  workflow  editor  GUI  to  enable  more  accurate  recording.  The  next  step  will  be  to  incorporate  more  IR 
procedures  into  the  workflow  editor  and  begin  collecting  data  on  those  procedures. 

Although  this  study  was  limited  to  workflow  recording  of  the  procedure  itself,  workflow  recordings  could  also  provide 
additional  benefit  by  expanding  the  scope  beyond  the  actual  procedure.  Following  the  flow  of  patient  along  with  the 
flow  of  the  procedure  is  a  critical  component  to  creating  a  greater  efficiency  not  only  in  the  procedure  room,  but  also  in 
the  perioperative  environment.  Many  of  the  inefficiencies  found  in  hospitals  today  involve  the  flow  of  patients,  which 
should  be  optimized  so  as  to  minimize  the  time  physicians  spend  waiting  for  the  next  patient,  and  minimizing  the  time 
that  patients  spend  waiting  for  the  physician.  Using  workflow  recording  and  analysis  at  the  departmental  level  may  aid 
in  revealing  inefficiencies  and  assessing  strategies  to  overcome  them. 
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Abstract —  Haptic  and  visual  displays  are  combined  to  realize 
cooperative,  force-feedback  tasks  over  the  internet.  The  opera¬ 
tors  ’’exert”  forces  on  a  virtual  object  which  in  turn  generates 
a  set  of  reaction  forces  to  be  displayed  on  the  haptic  devices.  A 
novel  cooperative  control  architecture  based  on  wave  variables 
is  implemented  to  realize  stable  operation  in  the  presence  of 
time  delays.  The  control  scheme  is  validated  experimentally  for 
a  manipulation  task  over  the  internet  using  a  pair  of  InMotion2 
robots.  Preliminary  results  are  also  presented  for  3D  tasks 
rendered  on  a  head-mounted  display  equipped  with  a  head 
tracker  for  changing  viewing  angles. 

I.  INTRODUCTION 

Robots  have  been  explored  as  possible  rehabilitation  aids 
in  laboratory  settings  for  well  over  a  decade.  These  investiga¬ 
tions  have  recently  expanded  into  the  field  of  ’’teletherapy,” 
whereby  a  clinician  can  interact  with  patients  in  remote 
locations  over  the  internet.  The  focus  of  our  research  is 
to  enable  cooperative  rehabilitative  tasks  in  virtual  environ¬ 
ments  intended  to  simulate  active  daily  living  tasks.  The  use 
of  a  ’’haptic”  (force-feedback)  device  in  conjunction  with 
a  video  display  will  allow  the  clinician  to  remotely  assess 
the  patient’s  condition  as  well  as  assist  the  patient  while 
performing  rehabilitation  tasks. 

This  article  describes  an  architecture  for  cooperative  re¬ 
habilitation  over  the  internet  using  wave  variables  in  a 
virtual  environment.  The  article  begins  with  a  brief  re¬ 
view  of  previous  work  on  cooperative  haptics  and  time- 
delay  compensation.  An  example  cooperative  beam  task  is 
presented  in  Section  V.  A  haptic/VR  interface  currently 
under  development  is  described  in  Sections  III  and  IV 
shown  in  Figure  1.  The  effects  of  internet  time  delay  and  a 
method  to  compensate  for  packet  jitter  is  described  in  Section 
VII.  Conclusions  on  the  implementation  of  the  cooperative 
rehabilitation  environment  are  given  in  Section  VIII  along 
with  some  directions  for  future  research. 

II.  PREVIOUS  WORK 

There  have  been  several  successful  demonstrations  of  the 
use  of  force-feedback  devices  for  rehabilitation.  Researchers 
at  the  University  of  Reading  have  incorporated  a  three 
degree-of-freedom  (DOF)  Haptic  Master  in  their  prototype 
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system  for  stroke  therapy  [1].  The  GENTLE/S  is  used  as 
an  orthosis  to  support  the  arm  and  has  a  haptic  display  for 
“reach- and- grasp”  tasks  inside  a  virtual  room.  The  use  of  a 
haptic  interface  allows  reach  and  grasp  movements  to  be  re¬ 
trained  by  either  assisting  movement,  or  directing  movement 
towards  a  specified  target.  The  virtual  environment  is  used 
to  replicate  the  subject’s  viewpoint  in  the  real  world,  where 
the  background  mats  and  shapes  are  used  to  help  perception 
of  movement  and  space. 


Fig.  1.  Top  view  of  InMotion 2  workstation  showing  four-bar  linkage,  direct 
drive  motors,  workstation,  and  flat  panel  display. 

Cooperative  control  using  haptic  devices  has  been  at¬ 
tempted  on  several  virtual  reality  platforms.  A  pair  of  2- 
DOF  master  manipulators  was  used  by  Harvard  investiga¬ 
tors  to  simulate  thumb  and  index  fingertip  contact  with  an 
object  during  a  peg-in-hole  insertion  task  [2],  [3].  Japanese 
researchers  using  a  pair  of  6-DOF  PHANToM  devices  to  sim¬ 
ulate  dual- arm  contact  with  a  steering  wheel  for  arm  motor 
control  training  [4].  A  pair  of  6-DOF,  parallel  mechanism 
force  displays  were  used  by  researchers  at  the  University  of 
Tsukuba,  Japan  to  perform  interactive  patient- therapist  tasks 
over  the  internet  [5].  Although  predictive  displays  were  used 
to  help  operators  adjust  for  up  to  3  sec  delays,  explicit  time- 
delay  compensation  was  not  implemented.  Several  investiga¬ 
tors  have  incorporated  explicit  time-delay  compensation  in 
the  force-feedback  loops  of  haptic  systems.  Wave  variables 
were  introduced  by  M.I.T.  investigators  for  a  variety  of 
master/slave  scenarios  with  widely  varying  time  delay  [6]. 
Passivity  control  was  implemented  by  researchers  at  the 
University  of  Washington  to  enable  stable  interaction  with 
virtual  environments  in  the  presence  of  time  delay  [7]. 
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III.  HARDWARE 

The  Imaging  Science  and  Information  Systems  (ISIS) 
Center  at  Georgetown  University  Medical  Center  has  re¬ 
cently  assembled  a  robot  rehabilitation  testbed  consisting  of 
a  pair  of  InMotion 2  Shoulder-Elbow  Robots  from  Interactive 
Motion  Technology,  Inc.  [8].  The  InMotion 2  (IM2)  Robot  is 
a  direct-drive,  four-bar  linkage  with  a  planar  workspace  of 
90  x  60  cm  and  maximum  continuous  force  output  of  30  N 
in  each  direction  (see  Figure  1).  The  handle  is  pinned  to  the 
distal  end  of  the  outboard  link,  providing  a  third,  unactuated 
degree  of  freedom.  The  low  friction  and  apparent  mass  at 
the  handle  (1.33  kg)  makes  the  IM2  Robot  ideally  suited 
as  a  haptic  device  for  conducting  physical  therapy  over  the 
internet.  These  two  characteristics  will  allow  the  clinician 
to  remotely  assess  the  patient’s  condition  as  well  as  help 
the  patient  perform  rehabilitation  tasks.  Implementing  force- 
feedback  in  the  presence  of  time-delay  is  key  to  realizing 
this  technology  over  the  internet  and  is  a  cornerstone  of  the 
control  architecture  presented  here. 

A.  Virtual  Reality  Equipment 

The  3D  graphical  environment  was  projected  on  a  head- 
mounted  display  as  shown  in  Figure  2.  The  SVGA  3D  Pro 
I-Glasses  have  a  resolution  of  800  600  pixels  and  run  at  60 
Hz,  the  same  rate  as  the  refresh  rate  of  the  video  card.  The 
I-Glasses  take  VGA  input  from  the  computer.  An  InterSense 
InterTrax2  headtracker  mounted  on  top  of  the  HMD  tracks 
the  orientation  of  the  operator’s  head  to  within  0.02  deg  about 
all  three  axes.  The  headtracker  is  an  inertia  measurement  unit 
with  triaxial  gyros  and  accelerometers.  It  is  connected  to  the 
computer  via  USB  and  supported  on  linux  with  open  source 
drivers. 


Fig.  2.  Subject  performs  virtual  beam  task  while  donning  3D  Pro  I-Glasses. 
The  vantage  angle  is  changed  by  using  output  from  an  InterTrax2  head 
tracker  mounted  on  the  brow  of  the  display. 


IV.  SOFTWARE 

The  InMotion 2  Shoulder-Elbow  Robot  is  controlled  via  an 
AMD  XP  1800  desktop  running  RTFinuxFree  on  RedHat 


7.2.  RTFinux  is  a  hard  real  time  operating  system  that  loads 
the  linux  kernel  as  a  module  in  the  real  time  kernel.  There  are 
a  total  of  three  processes  running  concurrently  to  calculate 
dynamics  for  realistic  haptic  feedback,  draw  a  3d  graphical 
environment,  and  communicate  scene  information  over  the 
internet.  Interprocess  communication  is  achieved  through 
shared  memory.  The  control  process  was  implemented  as 
a  hard  real-time  task  and  uses  approximately  1.2%  of  the 
CPU  time  at  a  rate  of  200Hz.  The  virtual  object  generator  is 
implemented  as  described  in  Section  V  in  the  control  process. 
Scene  information  was  sent  as  a  16  byte  dataset  using  the 
UDP  protocol. 


Fig.  3.  Interprocess  communication  is  through  shared  memory  for 
three  concurrent  processes  that  calculate  dynamics,  draw  3d  graphics,  and 
communicate  over  internet 

A.  User  Interface 

The  graphical  user  interface  (GUI)  shown  in  Figure  4 
was  built  with  the  Tool  Command  Fanguage/Toolkit  (Tcl/Tk) 
and  allows  the  user  to  change  parameters  for  the  virtual 
beam  task.  The  virtual  beam  must  stay  within  the  workspace 
boundaries  shown  in  orange.  Buttons  at  the  top  of  the  screen 
are  used  to  select  the  task  and  display  position  and  time 
delay.  Control  and  beam  parameters  are  set  using  the  sliders 
beneath.  In  the  current  configuration,  we  have  designated  the 
gravity  vector  to  point  in  the  sagital  plane  of  the  operator  so 
that  s/he  is  pushing  away  when  lifting  the  beam,  i.e.  toward 
the  screen  in  Figure  1.  A  reduced  gravitational  acceleration  of 
=3  m/s2  was  used  in  order  not  to  exceed  the  force  capacity 
of  the  robot. 

B.  3D  Graphics 

The  3D  graphical  environment  shown  in  Figure  5  was 
built  using  Silicon  Graphics  Open  Inventor  Toolkit  and  sits 
on  top  of  OpenGF  giving  a  higher  level  control  of  the  3D 
workspace.  The  visual  rendering  typically  takes  place  at 
about  10  Hz,  and  the  position  coordinates  of  the  virtual  object 
are  read  from  the  computer’s  shared  memory  at  every  cycle 
to  update  its  position  in  the  virtual  scene.  The  head- tracker 
driver  returns  a  quarternion  which  is  used  to  transform  the 
camera  viewpoint  in  the  Open  Inventor  scene  graph  such 
that  the  camera  view  changes  with  the  head  movements  of 
the  user.  At  the  moment,  we  are  only  able  to  run  the  3d 
graphics  with  a  refresh  rate  of  lOhertz  because  the  video 
card  is  an  outdated  Nvidia  TNT2.  We  anticipate  the  ability 
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Fig.  4.  Graphical  user  interface  developed  for  2D  virtual  beam  task. 


Fig.  5.  Graphical  scene  for  3D  cooperative  beam  task  generated  with 
Open  Inventor  Toolkit.  This  is  a  series  of  views  from  left  side  through  the 
head-mounted  display  as  subject  pans  his  head  right. 


to  run  graphics  at  much  higher  refresh  rates  when  we  find  a 
faster  video  card  supported  by  the  kernel. 

V.  Cooperative  Control 

The  technique  for  realizing  cooperative  control  for  per¬ 
forming  tasks  over  the  internet  is  through  a  virtual  object 
generator  as  shown  in  Figure  6.  In  this  scenario,  both  robots 
are  considered  ”masters”that  independently  interact  with  the 
virtual  object,  which  is  considered  the  ’’slave”.  The  virtual 
object  generator  (VOG)  applies  the  sensed  ’’interaction” 
forces  from  the  masters  and  then  calculates  the  dynamics 
response  of  the  object.  The  motion  of  the  object  at  each 
’’contact”  point  is  then  transmitted  back  to  each  master  where 
it  is  tracked  by  a  PD  controller. 

The  virtual  object  dynamics  are  calculated  via  a  separate 
process  on  one  of  the  master  arm  computers,  i  and  2  are 
time  delays  caused  by  computational  processing  and  internet 
transit  time.  If  the  object  dynamics  are  being  calculated 


Fig.  6.  System  diagram  for  cooperative  virtual  object  manipulation  over 
the  internet. 

on  the  Master  1  computer,  then  1  is  primarily  due  to 
the  computational  delay  for  the  VOG  process  (essentially 
zero),  and  2  is  the  time  delay  caused  by  internet  transit. 
To  maintain  a  truly  cooperative  task  and  avoid  evoking  a 
leader-follower  strategy,  an  artificial  time  delay  1  based  on 
a  moving  average  of  2  is  applied  to  the  control  computer 
hosting  the  virtual  object  process  (Master  1  in  this  case). 

An  example  of  a  cooperative  rehabilitation  task  is  shown 
in  Figure  7.  The  patient  and  therapist  “pick  up”  opposite 
ends  of  a  virtual  beam  by  grasping  the  handle  that  coincides 
with  their  end  of  the  beam.  Object  parameters  such  as  mass, 
length,  and  inertia  can  be  adjusted  to  correspond  to  real-life 
objects  using  a  graphical  user  interface  on  the  therapist’s 
computer.  As  the  object  is  “lifted,”  the  side  that  is  lower 
will  begin  to  feel  more  of  the  weight,  thus  stimulating  the 
participants  to  maintain  the  beam  in  a  horizontal  position. 
Also,  if  one  side  tugs  on  the  object,  the  other  side  feels  it, 
thus  encouraging  a  cooperative  strategy  to  lift  the  object.  The 
complete  dynamics  for  the  virtual  beam  can  be  found  in  [9]. 


mflSTERttl  IHfiSTeRK2 


Fig.  7.  Cooperative  manipulation  task  showing  haptic  masters  manipulating 
opposite  ends  of  a  “virtual”  beam  across  the  internet. 

There  are  ways  to  deal  with  instability  caused  by  time 
delay.  We  have  been  using  the  wave  variable  approach  to 
manipulate  objects  over  the  internet[10],  [11].  Instead  of 
sending  a  direct  velocity  or  force  command,  an  impedance 
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“wave”  command  is  issued  by  the  master,  and  then  an 
impedance  “wave”  is  reflected  by  the  slave.  How  much  of 
the  push  wave  is  reflected  depends  upon  the  impedance  on 
the  slave  side.  A  soft,  spring-like  environment  has  a  low  im¬ 
pedance  and  will  not  reflect  the  incoming  wave  as  greatly  as 
a  rigid  environment.  In  addition  to  its  impedance/admittance 
analog,  the  performance  of  the  wave  variable  controller 
degrades  only  gradually  with  increasing  time  delay  and  is 
thus  is  well-suited  to  the  internet.  When  using  the  wave 
variable  approach,  it  is  important  to  distinguish  between 
which  object(s)  is  the  “master”  and  which  is  the  “slave”.  In 
our  application,  the  virtual  object  is  regarded  as  the  “slave” 
and  is  acted  upon  by  virtual  forces  from  the  two  IM2  robots 
which  are  the  “masters”. 

The  control  architecture  for  the  cooperative  task  using 
admittance  control  with  wave  variables  is  shown  in  Figure  8. 
In  this  scenario,  each  haptic  master  is  interacts  independently 
with  the  virtual  object  which  is  considered  the  “slave”.  The 
virtual  object  process  then  decodes  the  wave  variables  from 
each  master  into  a  set  of  force  commands  to  be  imparted 
on  the  slave.  The  virtual  object  generator  calculates  the 
resultant  motion  at  each  of  the  “contact”  points  and  transmits 
it  via  wave  variable  back  to  the  master.  Each  master  control 
process  then  decodes  the  slave  wave  variables  into  desired 
velocities  at  its  “contact”  point  and  is  then  input  to  the 
master’s  tracking  controller. 


[raster  m 


111  ASTER  ft  2 


Fig.  8.  Admittance  control  architecture  implemented  with  wave  variables. 


VI.  EXPERIMENTS  WITH  COOPERATIVE  AND 
UNCOOPERATIVE  TASKS 

Two  tasks  were  performed  within  the  confines  of  the 
workspace  to  simulate  both  cooperative  and  noncooperative 
control.  The  GUI  in  Figure  4  was  used  to  activate  the 
control  process  and  change  parameters.  The  nominal  beam 
parameters  were  chosen  to  correspond  to  a  1  x  6  ft  (0.305 
x  1.829  m)  wood  (pine)  shelf  with  mass  ^=5.575  kg 
and  inertia  ^=6.215  kg-m2.  Full  gravitational  acceleration 
of  =10  m/s2  was  applied  to  the  shelf  in  these  tests  to 
give  as  realistic  a  simulation  as  possible.  The  control  gains 


were  increased  to  m=l  800  N/m  and  m=90  N/m/s  to 
accommodate  the  heavier  weight  of  the  beam. 

In  the  first  task,  the  subject  on  the  left  attempts  to  lift  their 
end  of  the  shelf  while  the  subject  on  the  right  holds  their  end 
fixed.  The  resulting  beam  angle  in  Figure  9  shows  the  shelf 
angle  becoming  growing  in  magnitude  as  the  left  end  lifts 
up  and  pivots  about  the  right  side  being  held  fixed  by  the 
uncooperative  subject.  The  forces  on  the  right  side  are  plotted 
in  Figure  10.  The  vertical  force  remains  relatively  constant 
at  approximately  half  the  shelf  weight,  but  the  force  in  the 
horizontal  direction  increases  as  the  subject  on  the  left  tries 
to  lift  the  beam  vertically  while  being  forced  to  follow  a 
circular  arc  by  the  uncooperative  subject  on  the  right  side. 


Fig.  9.  Beam  angle  during  uncooperative  lifting  task. 


Fig.  10.  Forces  on  right  side  during  uncooperative  lifting  task. 


In  the  second  task,  both  subjects  attempt  to  lift  their  ends 
simultaneously  while  holding  the  shelf  as  level  as  possible. 
The  shelf  angle  in  Figure  11  remains  reasonably  close  to 
zero  as  the  shelf  is  lifted  off  the  sawhorses.  The  forces  on 
the  right  side  are  shown  in  Figure  12.  The  vertical  force 
is  again  approximately  half  the  shelf  weight.  However,  the 
force  in  the  horizontal  direction  is  nearly  zero  since  both 
ends  are  now  lifting  together. 

VII.  EXPERIMENTS  WITH  TIME  DELAY 
A.  Effect  of  Time  Delay  on  Wave  Variables 

The  effect  of  an  increase  in  time  delay  on  the  wave 
variable  implementation  is  to  decrease  the  system’s  natural 
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Fig.  11.  Beam  angle  during  cooperative  lifting  task. 


Fig.  12.  Forces  on  right  side  during  cooperative  lifting  task. 


frequency.  The  “communications  stiffness”  com  is  given  by 
delay  \  thus,  the  wave  impedance  should  be  increased  in 
proportion  to  the  time  delay  to  maintain  system  bandwidth 
[6].  However,  the  time  delay  also  introduces  an  apparent 
mass  proportional  to  delay,  corn  —  delay ,  which  pro¬ 
duces  a  heavier  feel  at  the  handle  as  the  time  delay  (or 
wave  impedance)  increases.  Thus,  a  tradeoff  exists  in  wave 
impedance  between  maintaining  high  system  bandwidth  and 
low  inertia  at  the  handle. 

To  demonstrate  the  robustness  of  the  wave  variable  ap¬ 
proach  to  time  delays,  a  network  emulation  package  called 
NIST  Net  was  used  to  simulate  longer  internet  delays.  NIST 
Net  was  designed  at  the  National  Institute  of  Standards  and 
Technology  (NIST)  for  use  on  Linux  operating  systems.  The 
emulator  allows  controlled,  reproducible  experiments  with 
network  performance  sensitive  applications  in  laboratory 
settings.  In  the  time  delay  experiments,  the  beam  starts  out 
horizontally.  The  subject  on  the  left  side  raises  their  side 
until  the  beam  is  at  a  45  degree  angle,  while  the  subject 
on  the  right  side  maintains  position.  After  pausing  for  2-3 
seconds,  the  left  subject  continues  until  the  beam  is  vertical 
and  lets  the  right  subject  (now  below  the  left  subject)  support 
the  total  weight  of  the  beam.  After  this,  the  operator  on  the 
right  side  lifts  their  side  of  the  beam  and  mirrors  the  actions 
done  on  the  left  side  until  the  beam  is  horizontal. 

Simulation  results  for  constant  roundtrip  delays  of  250, 
500,  and  1000  msec  are  superposed  in  Figure  13  along  with 


the  zero  time  delay  case.  The  decrease  in  system  stiffness 
from  the  zero  delay  test  is  evidenced  by  the  lower  frequency 
oscillations  of  the  beam  angle  as  the  time  delay  gets  larger. 
In  addition,  the  apparent  mass  at  the  handle  increases  from 
approximately  0.5  kg  for  a  50  ms  roundtrip  delay  to  20  kg 
for  a  roundtrip  delay  of  1  sec  making  manipulation  of  the 
beam  even  more  difficult. 


t  (sec) 

Fig.  13.  Beam  angle  for  10  kg  mass  and  constant  time  delays  of  0,  250, 
500,  and  1000  ms  under  wave  variable  control. 

B.  Compensating  for  Packet  Jitter 

The  User  Datagram  Protocol  (UDP)  is  a  non- guaranteed, 
best-effort  datagram  delivery  protocol  and  offers  less  re¬ 
liability  than  TCP  in  exchange  for  lower  overhead  and 
speedier  delivery  of  data.  These  qualities  makes  it  useful 
for  applications  such  as  videoconferencing  and  streaming 
multimedia,  where  data  must  be  streamed  constantly.  An 
application  program  using  UDP  must  deal  directly  with  end- 
to-end  communication  problems  such  as  packet  retransmis¬ 
sion,  reordering,  packetization,  and  reassembly.  Figure  14 
shows  the  roundtrip  time  for  UDP  packets  between  the 
John  P  Murtha  Neuroscience  Center  (Johnstown,  PA)  and 
Georgetown  University  (Washington,  DC)  observed  over  a 
period  of  60  seconds.  Figure  15  shows  the  variability  of 
the  time  delay,  known  as  packet  jitter,  calculated  from  the 
difference  between  the  time  delay  of  each  sample.  The 
communications  frame  rate  is  fixed  at  100  Hz  (period  of  10 
ms)  however  the  data  shows  that  only  72%  of  all  incoming 
packets  arrive  within  10  ms  of  the  previous  packet.  We  are 
currently  developing  for  the  receiving  end  an  adaptive  jitter 
buffer  that  is  capable  of  maintaining  the  framerate  and  order 
of  most  packets.  All  packets  arriving  into  the  buffer  are  held 
for  a  specified  amount  of  time,  allowing  packets  that  arrive 
later  to  be  placed  in  the  correct  order  in  the  buffer.  Data 
from  the  buffer  is  fed  into  the  dynamics  process  at  a  fixed 
rate  of  100  Hz.  By  dynamically  changing  the  size  of  the 
buffer  based  on  the  amount  of  packet  jitter,  we  can  keep 
the  buffer  size  minimal  while  maintaining  a  certain  level  of 
jitter  compensation.  This  technique  reduces  packet  jitter  at 
the  expense  of  increasing  the  apparent  time  delay.  Figure  16 
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shows  the  distribution  of  packet  jitter  for  the  sample  data 
collected.  According  to  the  chart,  if  the  jitter  buffer  is  set  to 
20ms,  the  average  time  delay  is  increased  by  20ms  whereas 
86%  of  packets  arrive  in  correct  order  and  at  a  fixed  rate. 


Time  Delay 


Sample  (unit) 


Fig.  14.  Time  delay  of  packets  between  Washington,  DC  and  Johnstown, 
PA  over  60  seconds. 
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Fig.  15.  Packet  jitter  of  packets  between  Washington,  DC  and  Johnstown, 
PA 

VIII.  CONCLUSIONS 

The  focus  of  our  research  is  to  enable  cooperative  reha¬ 
bilitative  tasks  in  virtual  environments  using  force-feedback 
devices.  This  paper  has  outlined  a  strategy  for  cooperative, 
force-feedback  control  of  virtual  objects  over  the  internet 
based  on  an  admittance  control  architecture  using  wave 
variables.  The  wave  variable  approach  was  stable  for  a  wide 
range  of  time  delays  tested,  including  artificial  delays  of  over 
1  sec.  The  virtual  environment  was  created  using  the  Open 
Inventor  Toolkit  rendered  on  a  orientation  tracked  head- 
mounted  display.  Future  work  will  focus  on  generating  more 
realistic  3D  tasks  created  using  the  Open  Inventor  Toolkit 
aimed  at  improving  recovery  from  stroke  paralysis  and  other 
neurological  disorders. 
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ABSTRACT 


Electromagnetic  trackers  have  found  inroads  into  medical  applications  as  a  tool  for  navigation  in  recent  years.  Their 
susceptibility  to  interference  from  both  electromagnetic  and  ferromagnetic  sources  have  prompted  several  accuracy 
assessment  studies  in  past  years.  To  the  best  of  our  knowledge,  this  is  the  first  accuracy  study  conducted  to  characterize 
measurement  accuracy  of  an  NDI  AURORA  electromagnetic  tracker  within  a  CyberKnife  radiosurgery  suite. 
CyberKnife  is  a  frameless,  stereotactic  radiosurgery  device  used  to  ablate  tumors  within  the  brain,  spine  and  in  recent 
years,  the  chest  and  abdomen.  This  paper  uses  a  data  collection  protocol  to  collect  uniformly  distributed  data  points 
within  a  subset  of  the  AURORA  measurement  volume  in  a  CyberKnife  suite.  The  key  aim  of  the  study  is  to  determine 
the  extent  to  which  large  metal  components  of  the  CyberKnife  stereotactic  radiosurgery  device  and  robot  mount 
contribute  to  overall  system  performance  for  the  AURORA  electromagnetic  device.  A  secondary  goal  of  the  work  is  to 
determine  the  variation  in  accuracy  and  device  behavior  with  the  presence  of  ionizing  radiation  when  the  LINAC  is 
turned  on. 

Keywords:  electromagnetic  tracking,  CyberKnife,  Aurora,  radiosurgery,  accuracy  analysis. 


1.  INTRODUCTION 


The  use  of  electromagnetic  tracking  systems  (EMTS)  as  a  tracking  modality  for  medical  applications  has  grown  sharply 
in  recent  years.  This  has  in  part  been  due  to  the  miniaturization  of  tracking  sensor  coils  and  refinement  of  tracking 
algorithms.  Their  main  benefit  over  other  conventional  tracking  modalities  such  as  optical  trackers  is  their  lack  of 
dependence  on  line-of-sight  for  operation.  This  allows  for  greater  freedom  of  operation  for  the  surgeon  and  facilitates 
the  tracking  of  flexible  endoscopes  and  needles  within  the  body. 

The  main  drawback  of  EMTS  however,  is  their  susceptibility  to  electromagnetic  interference.  The  interferences  can  arise 
from  electrical  sources  and  ferromagnetic  metals  in  close  proximity  to  the  electromagnetic  tracker  [6].  Measurement 
errors  are  largely  idiosyncratic  in  nature  and  vary  greatly  between  environments.  Therefore,  quantification  of  these 
errors  can  be  performed  in  each  environment  prior  to  deployment  of  an  Image-Guided  Surgery  (IGS)  procedure,  by 
collecting  data  using  a  calibration  phantom  [1,  4,  5]. 

The  effectiveness  of  any  surgical  navigation  is  largely  dependent  on  the  accuracy  and  reliability  of  the  tracking  system. 
The  purpose  of  accuracy  studies  therefore,  is  to  quantify  the  measurement  errors  within  an  operating  volume  and 
determine  optimal  work  conditions.  For  this  reason,  accuracy  studies  for  medical  applications  of  EMTS  thus  far  have 
largely  been  for  interventional  applications  where  the  use  of  an  IGS  system  is  anticipated  [1,7]. 
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To  the  best  of  our  knowledge,  this  is  the  first  study  conducted  to  quantify  the  accuracy  of  an  EMTS  within  a  CyberKnife 
(Accuray  Inc.,  Sunnyvale,  CA)  suite.  The  CyberKnife  device  functions  as  an  ablative  tool  by  delivering  multiple  beams 
of  highly  focused  radiation  to  tumor  sites.  CyberKnife  is  a  non-invasive,  frameless,  stereotactic  radiosurgery  device, 
initially  used  for  the  treatment  of  largely  immobile  tumors  such  as  spine  and  brain.  The  device  consists  of  a  6-MeV 
medical  grade  linear  accelerator  (LIN AC)  mounted  on  a  robotic  arm  (KUKA,  Germany).  The  recent  addition  of  the 
Synchrony  system  enables  the  tracking  and  compensation  of  tumor  motion  due  to  respiration.  This  is  achieved  through 
the  use  of  a  stereo  X-ray  imager  and  a  FlashPoint  infrared  optical  tracker.  Infrared  markers  are  affixed  to  the  patient 
vest.  Small  (0.8mm  diameter)  gold  fiducials  are  implanted  percutaneously  near  the  tumor  site.  Prior  to  treatment  a 
correlation  model  between  internal  fiducials  and  external  IR  markers  is  generated.  The  robot  is  then  commanded  to 
follow  this  model.  During  treatment,  X-ray  images  are  acquired  every  30-40  seconds  to  verify  this  correlation  model. 

CyberKnife  is  the  only  radiosurgery  system  available  today  that  is  capable  of  targeting  tumors  with  respiratory  motion 
artifacts.  One  potential  issue  is  the  radiation  dose  from  the  periodic  X-ray  imaging  used  to  verify  internal  tumor  location. 
A  real-time,  non  ionizing,  method  of  continuously  tracking  internal  tumor  location  would  thus  be  desirable.  Current 
EMTS  are  capable  of  providing  measurements  in  real-time  and  could  potentially  satisfy  this  requisite  function,  provided 
they  are  able  to  offer  sufficient  accuracy  in  the  CyberKnife  environment. 

There  are  two  key  goals  for  this  study.  The  first  is  to  quantify  baseline  accuracy  of  an  EMTS  within  the  CyberKnife 
suite.  The  second  is  to  provide  a  formal  analysis  of  changes  in  measurement  accuracy  and  device  performance  in  the 
presence  of  ionizing  radiation  while  the  CyberKnife  system  is  active. 


2.  MATERIALS  AND  METHODS 


2.1  Measurement  device 

A  prototype  unit  of  the  NDI  AURORA  electromagnetic  tracking  device  (Northern  Digital  Inc.,  Waterloo,  Canada)  was 
evaluated  within  the  CyberKnife  suite.  The  device  consists  of  a  flat  field  generator  (FG)  that  creates  a  pulsed  AC 
magnetic  field.  The  FG  is  connected  to  a  system  control  unit  (SCU),  which  interfaces  to  the  computer  through  an  RS- 
232  serial  connection.  The  AURORA  device  has  a  manufacturer  quoted  RMS  positional  accuracy  of  0.8mm  and  RMS 
orientation  accuracy  of  0.3°.  The  quoted  accuracy  is  for  tests  conducted  in  a  ferromagnetically  clean  environment  and 
often  hard  to  repeat  in  a  typical  work  environment.  However,  we  have  found  device  accuracy  in  a  ‘clean’  office 
environment,  at  our  research  facility  void  of  any  metal  components  within  a  1  -meter  radial  distance  from  the  field 
generator,  to  be  1.5mm  RMS  positional  and  0.5°  RMS  orientation  errors.  Measurements  were  made  using  a  pair  of  5- 
DOF  (degree  of  freedom)  MagTrax  needles  (Traxtal  Technologies,  Bellaire,  TX,  USA)  with  embedded  sensor  coils.  An 
NDI  Optotrak  infrared  optical  tracking  device  was  used  also  to  collect  measurements  simultaneously  with  the  AURORA 
device.  Optotrak  records  measurement  with  3-DOF  and  has  an  RMS  position  accuracy  of  0.1mm.  Since  the  accuracy  of 
this  device  is  close  to  one  magnitude  better  than  AURORA  accuracy,  it  was  chosen  to  use  this  device  as  ground  truth. 

2.2  Calibration  phantom 

A  precision  tri-stage  motion  platform  consisting  of  3  linear  actuators  driving  lead  screws  along  3  orthogonal  axes  was 
used  for  the  data  collection.  The  motion  along  each  axis  is  accurate  to  0.05mm.  Since  the  motion  platform  is  largely 
made  of  aluminum,  a  sensor  mount  was  machined  out  of  plexiglass  to  mount  sensor  needles  and  IR  markers  at  the  distal 
end  of  the  arm.  A  control  program  was  written  to  move  the  motion  platform  along  a  predetermined  path  at  fixed 
intervals  and  poll  the  AURORA  and  Optotrak  devices  for  position  measurements  successively.  A  total  of  100 
measurement  cycles  from  each  device  were  logged  on  the  host  computer  at  each  data  collection  node.  A  data  collection 
volume  of  240  x  240  x  240mm  located  75mm  in  front  of  the  AURORA  FG  was  defined  for  the  data  collection.  Data 
collection  nodes  were  distributed  evenly  at  40mm  intervals  along  the  X,  Y  and  Z-axes.  Fig  1  provides  a  view  of  the  test 
setup  within  the  CyberKnife  suite. 
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Figure  1:  Test  setup  within  the  CyberKnife  suite 


2.3  CyberKnife  test 

A  total  of  1 0  baseline  tests  and  3  tests  with  the  CyberKnife  LIN  AC  beam  turned  on  were  conducted  around  the  robot 
isocenter.  The  CyberKnife  treatment  plan  devised  consisted  of  2  paths  and  was  typical  of  those  used  to  treat  liver 
tumors.  A  20mm  collimator  was  used  for  all  beam  ON  tests  and  the  robot  isocenter  used  as  the  target  for  beam 
convergence.  The  first  treatment  path  consisted  of  41  beams  delivering  a  total  of  804  MU  (monitor  units)  of  radiation 
and  the  second  path  consisted  of  22  beams  delivering  a  total  of  438  MU.  The  CyberKnife  device  is  calibrated  such  that 
1MU  of  radiation  is  equivalent  to  1  rad  (or  lcGy)  at  the  CyberKnife  collimator  tip.  All  beams  were  of  2s  duration  and 
the  entire  procedure  lasted  20  minutes. 

2.4  Position  error 

A  least  square  minimization  of  errors  technique  was  used  to  transform  data  points  from  the  AURORA  coordinate  space 
to  Optotrak  coordinate  space  [3].  Measurement  at  each  point  was  taken  as  the  average  of  the  100  data  samples.  Position 
error  was  defined  as  the  Euclidean  distance  between  transformed  AURORA  points  and  corresponding  measured 
Optotrak  points.  8  points  were  manually  selected  for  registration  based  on  proximity  to  FG  so  as  to  provide  low  initial 
error.  The  same  8  registration  points  were  used  for  all  tests. 
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2.5  Orientation  error 

Since  the  needles  are  fixed  firmly  within  the  mount,  the  relative  orientations  of  the  needles  theoretically  do  not  change 
during  a  test.  In  practice  however,  slight  variability  will  exist  and  for  purposes  of  this  study  have  been  characterized  as 
the  orientation  error.  The  orientations  at  the  8  registration  nodes  were  averaged  to  provide  a  reference  orientation.  At 
each  node,  orientation  error  was  then  evaluated  as  the  angular  difference  between  reference  orientation  and  the  average 
of  100  orientation  samples  at  the  specific  node.  In  addition,  the  variance  within  the  100  samples  at  each  data  node  was 
evaluated  as  a  measure  of  the  jitter  in  AURORA  measurements. 

2.6  Statistical  analysis 

Generalized  linear  models  were  utilized  for  repeated  measures  regression  to  investigate  the  relationships  between  both 
the  position  and  orientation  errors  and  field  generator  (FG)  to  sensor  separation  separately  for  each  error  measure  and  test 
condition  for  multiple  repeated  tests.  It  is  known  that  error  distribution  is  non-isotropic  for  EMTS  [2].  Therefore,  it  was 
chosen  to  model  the  errors  not  only  as  a  function  of  FG  to  sensor  separation,  but  also  as  a  function  of  position  along  the 
X,  Y  and  Z  axes.  The  natural  log  of  the  error  data  was  taken  prior  to  analysis  to  accommodate  the  assumption  of 
normally  distributed  errors  around  the  model  estimate.  The  regression  model  included  the  three  axes,  all  possible 
interactions  of  the  axes,  the  FG  to  sensor  separation,  and  an  indicator  variable  for  needle  1.  Additionally,  to  check  for 
consistency  across  experiments,  an  exploratory  analysis  was  done  examining  the  significance  and  parameter  direction  for 
each  test  separately.  Any  factor  was  considered  significantly  associated  with  error  measurement  if  the  p-value  for  the 
parameter  estimate  was  less  than  0.05.  P-values  indicate  whether  the  parameter  estimate  is  significantly  different  from  0, 
indicating  an  effect  of  that  variable  on  the  specified  measurement  error.  The  sign  of  the  parameter  estimate  is  an 
indication  of  whether  the  error  increases  or  decreases  with  increasing  values  of  the  variable. 


3.  RESULTS 


3.1  Baseline  results 

An  RMS  position  error  of  3. 1mm  and  average  position  error  of  2.7mm  were  computed  for  the  10  baseline  tests.  RMS 
orientation  error  was  evaluated  to  be  0.9°,  mean  orientation  error  of  0.7°.  Boxplots  of  position  and  orientation  errors  for 
the  10  tests  is  provided  in  Fig  2.  Fig  3  a  and  3b  provide  histograms  of  collective  position  and  orientation  errors  for  the 
two  needles  over  the  10  tests.  The  errors  were  shown  to  fit  a  log-normal  distribution. 
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3.2  LINAC  beam  ON  results 

RMS  position  error  of  2.3mm  and  average  position  error  of  1 .7mm  were  noted  for  the  3  baseline  tests.  RMS  orientation 
error  was  evaluated  to  be  0.9°  and  mean  orientation  error  to  be  0.6°.  Boxplots  of  the  position  and  orientation  errors  are 
provided  in  Fig  4a  and  4b  respectively.  Fig  5a  and  5b  provide  histograms  of  collective  position  and  orientation  errors  for 
the  two  needles.  The  reduced  position  and  orientation  errors  for  CyberKnife  beam  ON  tests  are  largely  explicable  due  to 
the  reduced  number  of  tests  (10  tests  for  baseline  as  opposed  to  3  tests  for  beam  ON). 
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Figure  4a:  Boxplot  summary  of  position  errors  for  LINAC 

Figure  4b:  Boxplot  of  orientation  errors  for  LINAC  beam  on 
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Figure  5  a:  Histogram  of  position  errors  for  LINAC  beam  on 
tests 


Figure  5b:  Histogram  of  orientation  errors  for  LINAC  beam 
on  tests 


3.3  Model  results 

Baseline  versus  LINAC  beam  ON  conditions  were  significantly  different  from  each  other  for  both  magnitude  and 
orientation  (p<0.0001  for  each).  The  overall  results  are  presented  in  Table  1  for  baseline  experiments  and  Table  2  for 
active  experiments.  This  difference  is  most  notable  in  the  Y  axis  since  the  direction  of  the  parameter  estimates  (positive 
versus  negative)  change  depending  on  the  condition  (baseline  versus  active). 

The  parameters  from  these  models,  displayed  in  Tables  1  and  2,  can  be  used  to  create  regression  equations  that  model  the 
position  and  orientation  errors  as  a  function  of  the  data  point  within  the  AURORA  measurement  volume,  FG  to  sensor 
separation  (dFG)  and  the  interactions  between  the  individual  axes.  The  regression  equation  defining  position  error  for 
baseline  (ErrIpos)  and  LINAC  (Err5/?05)  active  tests  are  provided  in  Equations  1  and  2  respectively.  The  needle  effect  is 
ignored  for  further  discussion  as  it  is  included  as  a  control  for  appropriate  estimation,  but  is  not  of  interest  as  a  condition 
affecting  accuracy. 

\n(ErrLpos)  =  -1.63  -  0.005X  -  0.0027  +  0.0004Z  +  0.009<iFG  +  0.06\Needle\  (Eq.  1) 

-  0.000007X7  -  0.000009XZ  -  0.00017Z  -  0.00000005X7Z 

\n(ErrBpos)  =  -2.242  +  0.003X  +  0.0087  +  0.00 1Z  +  0.0l2dFG  -  0.066Needlel  (Eq.2) 

-  0.00005X7  +  0.00002XZ  +  0.000017Z  -  0.0000008X7Z 

The  most  noticeable  trend  from  the  linear  model  analysis  was  the  reversal  of  the  effects  of  the  Y  axis  and  the  interaction 
of  Y  and  Z  axes  between  baseline  tests  and  LINAC  beam  on  tests.  This  shows  the  primary  differences  between  testing 
conditions.  Additionally,  there  was  decreased  significance  between  position  errors  and  Z-axis  locations  for  beam  ON 
tests  compared  to  baseline  tests.  In  contrast,  the  largest  parameter  estimates  were  consistently  for  FG  to  sensor 
separation,  regardless  of  test  type  or  error  type.  Note  that  each  measure  is  highly  statistically  significant.  Flowever,  the 
parameter  size  indicated  a  small  effect  of  each  parameter,  indicating  small  errors.  Table  3  presents  estimates  of  errors  for 
specific  values  of  X,  Y,  and  Z  for  needle  1 .  The  position  errors  range  from  (min)  to  (max),  all  of  which  are  very  small. 
Similarly,  the  orientation  errors  range  from  (min)  to  (max),  indicating  successfully  small  errors  throughout  the 
measurement  space. 
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For  the  analyses  examining  each  experiment  separately,  parameters  with  larger  p- values,  such  as  the  Z-  axis,  the 
significance  fluctuated  more  between  tests.  In  addition  a  distinct  sign  change  for  Y-axis  and  Y-Z-axis  interactions  was 
observed  between  baseline  and  beam  ON  trials,  indicating  that  the  effect  of  the  Y  axis  on  the  magnitude  of  the  errors  was 
reversed  in  the  Cyberknife  suite,  consistent  with  findings  in  the  overall  analyses  above.  In  contrast,  for  both  baseline 
tests  and  beam  ON  tests,  the  significance  for  FG  to  sensor  separation  was  strong  for  all  tests  and  the  estimate  always 
positive,  implying  that  an  increase  in  FG  to  sensor  separation  was  strongly  associated  to  an  increase  in  error. 


Table  1 :  Model  results  for  baseline  tests 


Position  Error 

Orientation  Error 

Parameter 

Estimate 

p-value 

Estimate 

p-value 

Intercept 

-1.630 

<0.0001 

-2.667 

<0.0001 

X-Axis 

-0.005 

<0.0001 

-0.007 

<0.0001 

Y-Axis 

-0.002 

<0.0001 

-0.002 

<0.0001 

Z-Axis 

0.0004 

0.0012 

0.006 

<0.0001 

FG  Distance 

0.009 

<0.0001 

0.012 

<0.0001 

Needle  =  1 

0.061 

<0.0001 

0.104 

<0.0001 

Interaction  X-Y-  Axes 

-0.000007 

<0.0001 

-0.000002 

0.29 

Interaction  X-Z-  Axes 

-0.000009 

<0.0001 

-0.00004 

<0.0001 

Interaction  Y-Z-  Axes 

-0.00001 

<0.0001 

-0.000008 

<0.0001 

Interaction  X-Y-Z-  Axes 

-0.00000005 

<0.0001 

-0.00000007 

<0.0001 

Table  2:  Model  results  for  beam  ON  tests 


Position  Error 

Orientation  Error 

Parameter 

Estimate 

p-value 

Estimate 

p-value 

Intercept 

-2.242 

<0.0001 

-2.403 

<0.0001 

X-Axis 

0.003 

<0.0001 

-0.009 

<0.0001 

Y-Axis 

0.008 

<0.0001 

0.008 

<0.0001 

Z-Axis 

0.001 

0.012 

0.012 

<0.0001 

FG  Distance 

0.012 

<0.0001 

0.016 

<0.0001 

Needle  =  1 

-0.066 

0.0008 

0.193 

<0.0001 

Interaction  X-Y-  Axes 

-0.00005 

<0.0001 

-0.00008 

<0.0001 

Interaction  X-Z-  Axes 

0.00002 

<0.0001 

-0.00005 

<0.0001 

Interaction  Y-Z-  Axes 

0.00001 

0.002 

0.000002 

0.49 

Interaction  X-Y-Z-  Axes 

-0.00000008 

0.003 

-0.0000002 

<0.0001 
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Table  3.  Comparison  of  estimated  and  actual  errors  from  model  equations  for  arbitrary  position  values. 


Variables 

Actual  Errors 

Estimated  Errors 

X 

Y 

z 

FG  distance 

Position 

Orientation 

Position 

Orientation 

175.48 

139.3 

-92.26 

242.31 

0.2 

0.4 

0.69 

0.42 

55.38 

21.88 

-135.51 

148.01 

0.62 

0.06 

0.60 

0.18 

54.17 

-139.27 

-134.92 

201.33 

0.7 

0.39 

1.08 

0.38 

185.52 

-18.19 

-371.73 

415.85 

5.43 

2.56 

5.20 

4.75 

-105.01 

102.03 

-220.65 

264.81 

2.54 

0.46 

2.77 

0.34 

4.  DISCUSSION 

Continual  refinement  of  electromagnetic  tracking  algorithms  and  sophistication  in  hardware  implementations  has  brought 
about  improvements  in  measurement  accuracy  and  reliability  for  electromagnetic  tracking  systems.  This  study  has 
evaluated  the  accuracy  of  an  EMTS  within  this  environment  under  the  influence  of  ionizing  radiation. 

The  results  of  this  study  indicate  that  while  there  are  noticeable  trends  in  AURORA  system  performance  with  and 
without  the  presence  of  ionizing  radiation,  the  overall  effect  on  measurement  accuracy  is  minimal.  Based  on  these 
results  the  AURORA  device  has  the  potential  to  work  well  within  the  CyberKnife  suite,  either  as  part  of  an  IGS 
application  or  as  a  means  for  direct,  continuous  tumor  tracking  during  a  stereotactic  radiosurgery  procedure  within  the 
CyberKnife  suite.  Further  studies  are  planned  to  verify  this  supposition. 
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ABSTRACT 

Lung  biopsy  is  a  common  interventional  radiology  procedure.  One  of  the  difficulties  in  performing  the  lung  biopsy 
is  that  lesions  move  with  respiration.  This  paper  presents  a  new  robotically  assisted  lung  biopsy  system  for  CT 
fluoroscopy  that  can  automatically  compensate  for  the  respiratory  motion  during  the  intervention.  The  system 
consists  of  a  needle  placement  robot  to  hold  the  needle  on  the  CT  scan  plane,  a  radiolucent  Z-frame  for  registration 
of  the  CT  and  robot  coordinate  systems,  and  a  frame  grabber  to  obtain  the  CT  fluoroscopy  image  in  real-time.  The 
CT  fluoroscopy  images  are  used  to  noninvasively  track  the  motion  of  a  pulmonary  lesion  in  real-time.  The  position 
of  the  lesion  in  the  images  is  automatically  determined  by  the  image  processing  software  and  the  motion  of  the  robot 
is  controlled  to  compensate  for  the  lesion  motion.  The  system  was  validated  under  CT  fluoroscopy  using  a 
respiratory  motion  simulator.  A  swine  study  was  also  done  to  show  the  feasibility  of  the  technique  in  a  respiring 
animal. 

Keywords:  lung  biopsy,  robot,  motion  tracking,  computer  integrated  surgery 


1.  INTRODUCTION 


1.1  Clinical  Significance 

Lung  biopsy  is  a  common  interventional  radiology  procedure.  The  purpose  of  the  procedure  is  to  obtain  a  tissue 
sample  from  a  suspicious  lesion  in  the  lung.  This  tissue  sample  is  then  analyzed  by  a  pathologist  to  determine  if 
cancer  is  present.  As  lung  cancer  screening  becomes  more  prevalent,  an  increasing  number  of  suspicious  lesions  will 
be  found,  and  more  lung  biopsies  will  be  required1. 

One  of  the  difficulties  in  lung  biopsy  is  that  lesions  move  with  respiration,  which  can  make  it  difficult  to  locate 
lesions  and  accurately  perform  the  biopsy.  CT  fluoroscopy  (CTF)  combines  the  advantages  of  both  CT  and 
fluoroscopy  in  that  moving  lesions  in  internal  organs  can  be  viewed  in  nearly  real-time,  allowing  compensation  for 
respiratory  motion  during  the  intervention.  It  is  also  possible  to  see  the  needle’s  interaction  with  the  target  lesions 
under  CTF,  resulting  in  improved  success  rate  of  the  percutaneous  biopsy2  (CTF  93.7%  vs.  CT  88.2%). 
Additionally,  CTF  decreases  procedure  time  by  at  least  a  factor  of  2  compared  with  conventional  CT  guided 
procedures3. 

Currently,  lung  biopsy  is  performed  freehand.  Based  on  preoperative  image  data,  the  physician  identifies  the  skin 
entry  point  and  the  lesion,  thus  defining  the  desired  needle  trajectory.  The  physician  then  aligns  the  needle  by  hand 
and  partially  inserts  it  towards  the  lesion.  The  physician  proceeds  with  further  insertion  of  the  needle,  checking  the 
position  of  the  needle  by  re-scanning  as  necessary.  The  main  problem  with  freehand  biopsy  is  that  the  physician’s 
accuracy  is  limited  when  initially  lining  up  the  needle  and  then  staying  on  course  throughout  the  procedure. 
Additionally,  when  the  physician  releases  the  needle,  the  needle  can  drift  or  tilt  away  from  the  desired  path.  In 
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response  to  these  problems,  we  propose  integrating  CTF  imaging  with  a  medical  robot  for  precise  placement  of  the 
needle. 

1.2  Prior  Technical  Developments 

For  robotically  assisted  needle  placement,  one  important  concept  is  the  remote  center  of  motion  (RCM)  which 
provides  rotational  motion  around  a  fixed  fulcrum  point  in  space.  Stoianovici  developed  a  two  degrees-of-freedom 
(DOF)  RCM  robot  with  a  radiolucent  needle  driver  for  percutaneous  renal  access  under  fluoroscopic  guidance4. 
Taylor  adapted  the  RCM  module  for  microsurgical  augmentation5. 

In  addition  to  the  hardware  developments,  many  robotic  systems  have  been  integrated  with  guiding  imaging 
modalities.  Hong6  presented  an  ultrasound-driven  needle-insertion  robot  for  percutaneous  cholecystostomy.  Loser7 
presented  a  system  for  needle  insertion  guided  by  visual  servoing  in  a  CTF  scanner.  Bascle  presented  an  approach 
for  needle  placement  under  X-ray  fluoroscopy8.  Susil9  developed  a  registration  method  using  a  Z-frame  for  needle 
insertion  inside  a  CT  scanner. 


2.  METHOD 


2.1  System  Components 

As  shown  in  figure  1,  the  main  system  components  include  a  CTF  scanner,  a  needle  insertion  robot  with  an 
associated  control  computer,  a  Z-frame,  a  frame  grabber,  and  an  image  processing  computer. 


Passive  Ann  Cartesian  Bridge 


Image  Processing  Computer 


Figure  1:  System  components 


The  CTF  scanner  is  a  Siemens  Somatom  Volume  Zoom.  The  maximum  frequency  of  the  CTF  scan  is  approximately 
6Hz.  The  needle  placement  robot  was  developed  by  the  Urology  Robotics  Group  at  Johns  Hopkins  Medical 
Institutions10.  It  includes  a  3-Degrees-Of-Freedom  (DOF)  Cartesian  motion  stage  mounted  on  the  CT  table  so  that 
the  robot  will  move  with  the  CT  table.  The  Cartesian  stage  is  connected  to  a  7-DOF  adjustable  unencoded  passive 
arm,  used  for  gross  positioning  of  the  needle  drive  stage.  The  rotation  stage  is  a  2-DOF  RCM.  The  needle  drive  stage 
is  a  1-DOF  friction  transmission  with  a  radiolucent  end-effector8  integrated  with  a  Z-frame  for  registration  of  the 
robot  and  CT9.  During  the  lung  biopsy  intervention,  CTF  images  are  obtained  as  needed  to  position  the  needle.  The 
real-time  display  from  the  CTF  monitor  is  captured  using  a  frame  grabber  (Accustream  170,  Foresight  Imaging, 
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Lowell,  Massachusetts,  USA).  The  frame  grabbed  images  are  processed  by  an  image  processing  computer  to 
estimate  the  real-time  position  of  the  target  lesion.  The  image  processing  computer  then  uses  the  inverse  kinematics 
of  the  robot  to  generate  control  commands  which  are  sent  to  the  robot  through  an  ethemet  cable  and  TCP/IP 
connection.  The  robot  then  moves  accordingly  to  compensate  for  the  respiratory  motion  of  the  patient. 


2.2  Clinical  Workflow 

The  envisioned  scenario  for  CTF-guided  robotically  assisted  lung  biopsy  is  as  follows: 

1.  The  patient  is  scanned  under  breath  holding  to  obtain  a  3D  preoperative  CT  volume  at  the  end-of- 

inspiration. 

2.  The  physician  selects  the  entry  and  target  locations  of  the  biopsy  path  in  the  CT  volume. 

3.  The  physician  starts  the  CTF  scan,  allowing  the  system  to  track  the  real-time  position  of  the  target 

lesion. 

4.  Patient’s  skin  at  the  entry  point  is  anesthetized  with  1%  Lidocaine. 

5.  The  robot  moves  the  needle  to  the  entry  point,  aligns  the  needle  with  the  path,  and  inserts  the  needle 
into  the  target  lesion  based  on  the  motion  tracking  result. 

6.  The  robot  releases  the  needle. 

7.  The  physician  manually  inserts  the  needle  under  the  guidance  of  the  needle  holder  of  the  robot. 

8.  The  physician  injects  the  therapeutic  agent  or  takes  the  biopsy  sample. 

In  order  to  increase  safety,  the  physician  supervises  the  needle  insertion  at  the  robot  control  computer.  The  system 
can  be  halted  at  any  time  by  the  physician.  Before  the  needle  enters  the  body,  the  robot  can  make  quick  adjustments 
of  the  needle  pose  based  on  the  real-time  position  of  the  target  lesion.  The  patient  holds  their  breath  during  the 
needle  insertion  to  ensure  accurate  targeting  and  to  avoid  potential  needle  bending.  Once  the  needle  is  in  the  lesion, 
the  robot  should  release  the  needle  so  that  the  patient  can  breathe  again  and  the  needle  can  move  freely  with 
respiration.  The  current  needle  driver  does  not  allow  us  to  release  the  needle,  but  a  new  version  of  the  needle  driver 
is  being  designed  to  include  this  feature. 


2.3  Three  Dimensional  Motion  Tracking  of  Pulmonary  Lesions 

In  lung  biopsy,  the  position  of  the  target  lesion  may  vary  due  to  intrinsic  causes  such  as  respiratory  motion  or 
extrinsic  reasons  such  as  interactions  between  the  tissue  and  a  surgical  tool.  The  positioning  information  of  the 
lesion  obtained  from  the  preoperative  CT  scan  may  not  be  correct  during  the  intervention.  Therefore,  it  is  necessary 
to  track  the  lesion’s  motion  in  3D  space.  CTF  can  be  used  for  motion  tracking  in  that  it  allows  the  tissue  inside  the 
imaging  plane  to  be  viewed  in  real-time.  In  order  to  estimate  both  in-plane  and  out-of-plane  motion  of  the  lesion,  the 
patient  needs  to  be  positioned  carefully  on  the  CT  table  so  that  the  lesion  moves  within  a  certain  distance  from  the 
imaging  plane  during  respiration.  This  is  feasible  because  respiratory  motion  is  limited  to  a  few  centimeters, 
allowing  the  off-plane  motion  of  the  target  lesion  to  be  estimated  using  the  lung  tissue  inside  the  imaging  plane.  The 
motion-tracking  algorithm  is  based  on  minimizing  the  Zero-mean  Sum  of  Squared  Differences  (ZSSD)  between  a 
reference  CTF  image  region  and  a  corresponding  region  in  the  preoperative  CT  volume.  The  relative  position 
between  the  target  and  the  reference  CTF  region  can  be  obtained  in  the  preoperative  CT  volume,  which  can  be  used 
to  estimate  the  lesion’s  real-time  position  with  respect  to  the  CTF  imaging  plane.  The  details  of  this  tracking 
algorithm  are  described  in  [1 1]. 


2.4  Registrations  and  Coordinate  Transformations 

The  goal  of  this  system  is  to  insert  the  needle  into  the  target.  The  real-time  transformation  between  the  needle  and 
the  target  can  be  calculated  as  following: 

77  Needle  77  Needle  _  77  Z- frame  _  77  CT  ^  tjCTF  >. 

tTWget  ~b Z- frame  *  ^ CT  *  b CTF  *  t T Wget  (!) 

where  Ff  represents  a  4x4  homogeneous  transformation  matrix  from  the  source  coordinate  system  A  to  the  target 
coordinate  system  B. 


Page  97 


Proceedings  of  SPIE  Medical  Imaging  2006:  Visualization,  Image-Guided  Procedures,  and  Display 

From  the  right  of  to  the  left  of  equation  (1),  F^fget  contains  the  real-time  position  of  the  lesion  with  respect  to  the 

CTF  imaging  plane,  obtained  from  the  motion-tracking  algorithm11;  consists  of  the  fixed  translation  from  the 

origin  of  the  frame  grabbed  CTF  image  to  the  origin  of  the  preoperative  CT  volume,  obtained  from  a  translation 
registration  (3 -DOF)  of  a  static  rigid  body  in  both  images.  Since  the  Z-frame  is  attached  to  the  robot,  F^~frame  is  a 

fixed  transformation  from  the  CT  space  to  the  robot  space.  The  Z-frame  contains  three  “Z”s  that  are  formed  by 
seven  radiolucent  fiducial  rods  located  at  known  positions.  As  shown  in  figure  2,  a  single  cross  sectional  CT  image 
allows  the  rigid-body  transformation  between  the  CT  and  the  Z-frame  to  be  determined7.  FNeefe  is  the  fixed 

^  J  Z- frame 

transformation  from  the  Z-frame  to  the  needle  space. 


(a)  Z-frame  (b)  A  CT  slice  of  Z-frame  (c)  Registered  Z-frame 

Figure  2:  Registration  of  Z-frame  to  CT 


Since  the  RCM  and  Cartesian  stages  are  connected  by  a  passive  arm,  they  need  to  be  registered  to  the  CT  space 
separately.  The  RCM  is  registered  through  the  attached  Z-frame.  Since  the  Cartesian  stage  is  rigidly  attached  to  the 
CT  table  using  a  curved  mount  and  nylon  straps,  (figure  3. a),  it  is  considered  to  be  aligned  with  coordinate  system  of 
the  CT  scanner.  This  assumption  is  reasonable  because  respiratory  motion  is  limited  to  a  few  centimeters.  In  such  a 
small  motion  range,  the  motion  error  of  the  robot  introduced  by  any  misalignment  of  the  Cartesian  stage  and  CT 
table  is  negligible.  To  move  the  needle  from  its  current  needle  position  to  the  target  position,  the  motion  of  the  robot 
can  be  calculated  from  f^eedle  and  the  inverse  kinematics  of  the  robot. 

T  arg  et 


2.5  Compensation  for  the  Motion  of  Target  Lesion 

To  compensate  for  the  motion  of  the  target  lesion,  a  control  loop  is  executed  in  the  software.  The  needle  is  retracted 
from  the  RCM  center  to  avoid  any  potential  collision  with  the  patient  during  the  tracking.  The  following  steps  are 
executed  inside  the  loop: 

1 .  Process  the  CTF  image  to  obtain  the  real-time  position  of  the  target  lesion. 

2.  Update  the  biopsy  path  using  the  result  of  motion  tracking. 

3.  Apply  the  inverse  kinematics  of  the  robot  to  update  the  needle  pose  according  to  the  new  path. 

This  is  a  typical  control  strategy  that  combines  the  visual  sensing  and  manipulation  in  an  open  loop  fashion, 
“looking  then  moving”5,  in  which  the  CTF  is  the  camera  or  the  eye  fixed  on  the  ground.  As  a  prototypical  system, 
the  skin  entry  point  is  currently  assumed  to  be  independent  from  respiration.  Therefore,  only  the  needle  orientation 
needs  to  be  updated  to  compensate  for  respiratory  motion.  In  reality,  the  motion  of  the  skin  entry  point  may  not  be 
negligible.  Although  both  the  hardware  and  software  of  the  system  are  capable  of  compensating  for  the  lesion’s 
motion  in  nearly  real-time,  for  safety  reasons,  we  envision  that  in  any  clinical  use  the  needle  will  be  manually 
inserted  by  the  physician  after  the  needle  is  aligned.  Particularly  in  lung  biopsy,  automatic  needle  insertion  using 
robot  may  increase  the  chance  of  pneumothorax.  Therefore,  the  patient  is  asked  to  hold  his  or  her  breath  during  the 
insertion  to  avoid  potential  needle  bending.  The  needle’s  position  is  automatically  adjusted  only  before  it  enters  the 
patient. 
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3.  EXPERIMENTS  AND  RESULTS 


To  test  the  system’s  performance,  both  phantom  and  swine  studies  were  conducted  at  Georgetown  University 
Medical  Center  on  a  Siemens  Volume  Zoom  CT/CTF  scanner.  The  frame  rate  of  the  CTF  scans  was  6  Hz. 


3.1  Phantom  Study 

The  accuracy  of  the  system  was  evaluated  in  a  phantom  study  using  a  respiratory  motion  simulator.  Since  the 
motion-tracking  algorithm  relies  heavily  on  the  texture  of  the  lung,  we  developed  a  lung  phantom  with  similar 
texture  information.  The  phantom  was  made  from  seashells  and  rice  packaged  in  a  tennis  ball  container  as  shown  in 
figure  3.b.  The  phantom  was  mounted  on  a  3D  translational  stage  which  held  the  phantom  on  the  CTF  scan  plane 
and  could  be  computer  controlled  to  simulate  any  motion  pattern.  This  3D  translational  stage  was  originally 
developed  to  serve  as  a  respiratory  motion  simulator  during  CyberKnife  radiotherapy  motion  compensation 
experiments12. 


(b)  CT  image  of  lung  phantom  (c)  Back  view  of  phantom  study 

Figure  3:  Phantom  study 


Before  the  experiments,  the  motion  simulator  was  manually  aligned  with  the  CT  table.  Several  1  mm  fiducials  were 
attached  to  the  surface  of  the  phantom.  The  phantom  was  CT  scanned  with  a  1  mm  slice  thickness  and  0.74  mm 
pixel  size.  A  fiducial  on  the  phantom  surface  was  selected  as  the  needle  target.  The  needle  entry  point  was  selected 
to  be  in  the  air  above.  After  the  system  was  initialized,  the  phantom  was  moved  by  the  respiratory  motion  simulator 
from  its  original  position  to  some  other  known  position.  A  CTF  image  at  the  new  position  was  obtained.  The  frame 
grabbed  CTF  image  was  processed  to  obtain  the  position  of  the  target  lesion.  The  robot  then  moved,  aligned  and 
drove  the  needle  to  the  target.  The  difference  between  the  target  and  the  final  robot  position  was  measured  to  obtain 
the  accuracy  of  the  system. 

To  measure  the  system  error,  the  encoders  of  the  robot  were  used  as  a  precision  measuring  device.  The  robot  was 
manually  moved  using  its  translational  stage  until  the  needle  tip  touched  the  target  fiducial.  The  amount  the  robot 
had  to  be  moved  was  recorded  as  the  system  error  as  shown  in  Table  1.  The  tracking  error  was  also  estimated  in  this 
study  by  comparing  the  motion  amplitude  detected  by  the  tracking  algorithm  to  the  programmed  amplitude  (Table 

i). 
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_ Table  1:  Experimental  Results _ 


Parameter 

Measured 

Number 
of  Trials 

Average 

Error 

Variance 
of  Error 

System  error 

10 

1.7  mm 

0.5  mm 

Tracking  error 

10 

0.4  mm 

0.3  mm 

Although  the  motion  in  the  above  study  was  not  continuous,  comparable  results  were  observed  in  another  study 
under  continuous  phantom  motion.  As  shown  in  figure  4,  the  respiratory  motion  simulator  was  commanded  to  move 
in  sinusoid  curves  along  the  three  principal  axels  simultaneously.  Since  no  synchronization  was  used  in  the  study, 
the  results  of  motion  tracking  were  fit  to  the  ground  truth  of  the  programmed  motion. 


(a)  Left-Right  Motion  (b)  Anterior-Posterior  Motion  (c)  Superior-Inferior  Motion 

Figure  4:  Results  (blue  dots)  of  motion  tracking  in  mm.  Red  lines  are  the  ground  truth. 


3.2  Swine  Study 

The  system  was  also  tested  in  animal  studies  under  an  approved  animal  care  protocol,  with  endotracheal  intubation 
and  under  general  anesthesia.  A  3D  CT  scan  was  taken  with  the  ventilator  held  in  end  respiratory  phase  of  the  tidal 
volume  breathing  cycle.  To  avoid  the  risk  of  inducing  a  pneumothorax  and  compromising  the  animal,  no  fiducial 
markers  was  implanted  in  the  lung.  Instead,  an  anatomical  location  of  the  lung  was  identified  in  the  CT  volume  as 
the  target  point.  During  the  treatment,  the  swine  was  mechanically  ventilated.  The  CTF  images  were  used  to  track 
the  motion  of  the  target  point.  At  a  certain  respiratory  phase,  the  ventilator  was  stopped  to  simulate  the  breath 
holding  of  the  swine.  The  end-effector  was  aligned  based  on  the  motion-tracking  algorithm;  then,  the  needle  was 
pushed  into  the  lung  by  hand.  Figure  5. a  shows  one  of  the  animal  studies.  Figure  5.b  shows  the  planned  path  in  the 
preoperative  CT  volume,  and  figure  5.c  shows  the  actual  needle  path.  Although  there  was  no  numerical  ground  truth 
to  validate  the  needle  insertion,  good  correspondence  between  the  two  paths  was  observed  from  the  images,  meaning 
that  the  system  was  able  to  accurately  align  the  needle.  While  this  is  just  a  preliminary  result,  this  gives  us 
confidence  that  the  system  can  track  the  respiratory  motion  of  the  lung,  and  further  animal  studies  are  planned. 


(b)  Planned  needle  path 
Figure  5:  Swine  study 


(c)  Actual  needle  path 


(a)  Swine  in  CTF 
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4.  CONCLUSION  AND  DISCUSSION 


In  this  paper,  we  presented  the  feasibility  of  using  an  image-guided  surgical  robot  to  assist  lung  biopsy.  The  system 
integrates  a  real-time  non-invasive  tracking  algorithm  to  detect  the  motion  of  a  pulmonary  lesion.  An  open-loop 
control  is  used  to  automatically  compensate  for  the  target’s  motion.  Initial  experimental  results  showed  that  the 
biopsy  system  worked  well  with  a  respiratory  motion  simulator  and  had  reasonable  performance  in  a  swine  study. 
There  are  currently  no  other  published  systems  for  noninvasive  real-time  motion  compensation  of  pulmonary 
tumors. 

While  the  experiments  validated  the  overall  system  concept,  several  weaknesses  were  identified  that  need  to  be 
addressed.  First,  we  did  not  achieve  a  sufficient  working  volume,  due  to  interference  among  the  relatively  large  Z- 
frame,  the  RCM  and  the  patient.  We  could  not  reach  all  clinically  significant  locations  in  the  phantom  and  swine 
studies,  especially  considering  oblique  angles.  The  robot’s  passive  arm  had  to  be  very  carefully  manipulated  to 
ensure  that  not  only  the  Z-frame  stayed  in  the  CTF  filed  of  view  but  also  the  RCM  didn’t  collide  on  the  patient. 
Therefore,  the  RCM  and  the  Z-frame  need  to  be  redesigned  to  allow  for  more  clearance  for  the  robot  during  needle 
placement.  Second,  the  robot  was  rather  sluggish  in  following  the  target’s  motion.  We  must  note,  however,  that 
typical  surgical  robots  are  not  supposed  to  perform  quick  motions,  in  order  to  reduce  potential  injury  to  the  patient. 
Finally,  the  motion-tracking  algorithm  is  only  accurate  when  the  target  is  close  to  the  CTF  imaging  plane13.  The 
algorithm  is  not  robust  for  lung  regions  that  don’t  have  enough  texture. 

In  addition  to  improving  the  tracking  algorithm’s  robustness  and  accuracy,  future  work  will  include  developing  a 
new  needle  driver  so  that  the  needle  can  be  automatically  released  from  the  needle  driver  once  the  needle  has 
reached  the  desired  position14.  As  a  safety  feature,  it  would  also  be  desirable  to  integrate  force  sensing  in  the  needle 
driver  so  that  the  needle  could  be  released  when  abrupt  motion  of  the  patient  is  detected  by  processing  real-time  CTF 
image.  Finally,  the  real-time  needle  position  can  also  be  obtained  by  tracking  the  Z-frame  in  the  CTF  image.  Using  a 
method  such  as  visual  servoing,  the  needle  position  can  be  automatically  detected  and  used  to  adjust  the  needle’s 
pose  with  respect  to  the  target  lesion.  The  control  loop  of  the  system  can  then  be  closed,  which  is  called  “dynamic 
look-and-move”7. 
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ABSTRACT 

Lung  cancer  screening  for  early  diagnosis  is  a  clinically  important  problem.  One  screening  method  is  to  test  tissue 
samples  obtained  from  CT-fluoroscopy  (CTF)  guided  lung  biopsy.  CTF  provides  real-time  imaging;  however  on 
most  machines  the  view  is  limited  to  a  single  slice.  Mentally  reconstructing  the  direction  of  the  needle  when  it 
is  not  in  the  imaging  plane  is  a  difficult  task.  We  are  currently  developing  3D  visualization  software  that  will 
augment  the  physician’s  ability  to  perform  this  task.  At  the  beginning  of  the  procedure  a  CT  scan  is  acquired 
at  breath-hold.  The  physician  then  specifies  an  entry  point  and  a  target  point  on  the  CT.  As  the  procedure 
advances  the  physician  acquires  a  CTF  image,  at  breath-hold;  the  system  then  registers  the  current  setup  to  the 
CT  scan,  enabling  comparison  between  the  plan  and  current  situation.  As  the  CT  and  CTF  data  are  acquired 
at  different  breath-holds  we  expect  them  to  exhibit  displacements  of  up  to  4mm.  To  assess  the  performance 
of  different  registration  algorithms  for  CTF/CT  registration  we  propose  to  use  simulated  CTF  images.  These 
images  are  created  by  deforming  the  original  CT  volume  and  extracting  a  slice  from  it.  Realistic  deformation  of 
the  CT  volume  is  achieved  by  using  positional  information  from  electromagnetically  tracked  fiducials,  acquired 
throughout  the  respiratory  cycle.  To  estimate  the  dense  displacement  field  underlying  the  sparse  displacement 
field  provided  by  the  fiducials  we  use  radial  basis  function  interpolation.  Finally,  we  evaluated  Thirion’s  ’’demons” 
algorithm,  as  implemented  in  ITK,  for  the  task  of  slice-to- volume  registration.  We  found  it  to  be  unsuitable  for 
this  task,  as  in  most  cases  the  recovered  displacements  were  less  than  50%  of  the  original  ones. 

Keywords:  Image-Guided  Therapy,  CT-Fluoroscopy,  deformable  registration,  simulated  respiration,  radial  basis 
functions 


1.  INTRODUCTION 

Lung  cancer  is  a  leading  cause  of  death  in  the  United  States  with  an  overall  cure  rate  (five  year  survival)  of 
only  about  15%. 1  Lung  cancer  screening  has  the  potential  of  leading  to  early  diagnosis  and  treatment,  thus 
increasing  chances  of  survival.2  One  screening  method  is  to  test  tissue  samples  obtained  from  CT-Fluoroscopy 
(CTF)  guided  lung  biopsy. 

The  outcome  of  this  approach  is  highly  dependent  on  the  ability  of  the  physician  to  accurately  place  the 
biopsy  needle  within  the  suspected  nodule  based  on  the  CTF  images.  CTF  provides  the  physician  with  real-time 
imaging,  however  on  most  machines  it  is  limited  to  a  single  slice.  Mentally  reconstructing  the  direction  of  the 
needle,  when  it  is  not  in  the  imaging  plane,  using  a  single  slice  as  feedback  can  be  a  difficult  task,  even  for 
experienced  physicians. 

We  are  currently  developing  3D  visualization  software  that  may  augment  the  physician’s  ability  to  perform 
this  task.  The  system  works  as  follows.  At  the  beginning  of  the  procedure  a  CT  scan  is  acquired  at  breath- 
hold.  The  physician  then  specifies  an  entry  point  and  a  target  point  on  the  CT.  As  the  procedure  advances, 
the  physician  acquires  a  CTF  image  at  breath-hold;  the  system  then  registers  the  current  setup  to  the  CT  scan, 
enabling  comparison  between  the  plan  and  current  situation.  As  the  CT  and  CTF  data  are  acquired  at  different 
breath-holds  we  need  to  deformably  register  them. 

E-mail:  {zivy,stenzel,cleary}@isis. georgetown.edu 
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As  the  whole  system  depends  on  accurate  CTF/CT  registration  we  first  need  to  evaluate  the  performance 
and  limitations  of  different  deformable  registration  algorithms.  We  propose  to  use  a  simulation  framework  for 
respiratory  based  motion,  allowing  us  to  validate  registration  results  with  regard  to  a  known  transformation.  To 
simulate  a  CTF  image  we  extract  a  slice  from  a  second  volumetric  data  set  whose  transformation  with  regard 
to  the  original  volume  is  known.  Three  possible  methods  to  obtain  a  second  volumetric  data  set  with  known 
deformations  are:  (1)  Ad-hoc  methods;  (2)  Registration  based  methods;  and  (3)  Finite  element  methods  (FEM). 

Ad-hoc  methods  receive  as  input  a  volumetric  data  set  and  a  sparse  displacement  field  describing  the  defor¬ 
mation.  They  output  a  dense  displacement  field  and  a  corresponding  deformed  volume.  The  sparse  displacement 
field  is  generated  by  manually  displacing  a  small  number  of  points.  The  points  are  usually  anatomical  landmarks 
with  the  displacement  field  generated  via  interpolation3  or  approximation.4  Points  may  also  be  the  intersections 
of  a  regular  grid  that  overlaps  the  image  with  the  displacement  field  generated  via  free- form  deformations.5  The 
popularity  of  this  approach  is  due  to  its  simplicity,  and  that  it  does  not  require  additional  data  (i.e.  only  the  orig¬ 
inal  volume  is  required).  Unfortunately,  as  the  approach  is  not  constrained,  physically  impossible  deformations 
are  easily  generated. 

Registration  based  methods  receive  as  input  multiple  volumetric  data  sets  under  different  deformations.  They 
output  a  dense  displacement  field  describing  the  deformation  between  pairs  of  volumes.6, 7  This  approach  does 
not  incorporate  bio-mechanical  knowledge,  but  works  well  if  the  deformation  between  the  registered  data  sets 
is  smooth.  Applying  this  approach  to  structures  that  deform  considerably  requires  acquisition  of  multiple  data 
sets  throughout  the  deformation  process  (e.g.  a  set  of  MRI  volumes  acquired  between  end  expiration  and  end 
inspiration).  Note  that  using  the  displacement  field  generated  by  this  approach  as  a  gold  standard  can  be  biased 
if  the  same  algorithm  is  used  for  volume-to- volume  and  slice-to- volume  registrations. 

Finite  element  methods  receive  as  input  a  volumetric  data  set,  a  segmentation  of  the  anatomical  structures 
according  to  the  different  tissue  types,  and  the  bio-mechanical  properties  of  the  tissue  types.  They  output  a  dense 
displacement  field  and  a  corresponding  deformed  volume.  These  methods  have  have  been  studied  extensively, 
mainly  in  the  context  of  surgical  simulation,8, 9  and  registration.10, 11  This  approach  is  considered  the  most 
physically  accurate,  and  has  been  used  to  validate  free- form  registration  of  mammographic  MRI  data.12 

Our  approach  to  simulating  deformation  due  to  respiratory  motion  can  be  classified  as  a  constrained  version 
of  the  ad-hoc  approach.  We  deform  the  original  volumetric  data  based  on  the  motion  of  a  sparse  set  of  points, 
electromagnetically  tracked  fiducials,  which  are  acquired  from  organ  motion  of  a  respiring  animal.  As  is  the 
case  with  the  ad-hoc  approach,  computing  the  deformation  is  still  an  under-constrained  problem.  The  difference 
is  that  now  the  deformations  are  driven  by  physical  measurements  and  not  arbitrary  displacements.  Figure  1 
classifies  the  different  approaches  according  to  their  computational  complexity  and  physical  realism. 

Similar  approaches  to  our  own  have  been  recently  described  in  the  context  of  real-time  deformation  update 
of  a  CT  volume.13, 14  Both  papers  estimate  deformations  of  small  volumes  using  electromagnetically  tracked 
fiducials.  In13  the  deformation  field  is  obtained  using  a  variational  approach,  with  a  regularization  term  that 
enforces  smoothness.  No  quantitative  results  are  given  and  the  weighing  of  the  regularization  term  is  unspecified. 
In14  the  deformation  is  assumed  to  be  affine  which  is  plausible  for  very  small  volumes. 


Computational  Complexity 
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Figure  1.  Classification  of  approaches  for  creating  a  ground  truth  displacement  field  and  deformed  volumetric  data  set. 
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2.  MATERIALS  AND  METHODS 

To  assess  the  accuracy  of  deformable  slice-to- volume  (CTF/CT)  registration  we  have  developed  a  simulation 
based  framework  for  respiratory  based  motion,  providing  a  ground  truth  to  which  registration  results  can  be 
compared.  Our  method  is  based  on  simulating  CTF  images  using  a  CT  scan  acquired  at  end  expiration  that  is 
deformed  according  to  data  obtained  from  electromagnetically  tracked  fiducials. 

2.1.  Data  simulation 

Simulating  CTF  images  at  different  stages  of  the  respiratory  cycle  is  a  four  step  process:  (1)  Acquire  a  CT  scan 
and  positional  information  from  electromagnetically  tracked  fiducials;  (2)  Compute  a  deformation  field  based  on 
the  electromagnetic  tracking  data,  and  segment  the  lung  region  in  the  CT.  (3)  Deform  the  original  volume  in 
the  region  of  the  lungs  using  the  segmentation  and  deformation  field  from  the  previous  step.  (4)  Extract  a  slice 
from  the  deformed  volume,  this  is  our  simulated  CTF  image. 

Data  acquisition  proceeds  as  follows.  First,  four  magnetically  tracked  fiducials  are  placed  in  the  animal’s  liver 
(Fig.  2(a)),  under  an  approved  protocol,  and  a  CT  is  acquired  at  end  expiration.  Then  the  animal  is  transferred 
to  an  environment  where  there  are  no  ferromagnetic  materials  and  positional  data  from  the  fiducials  is  acquired 
throughout  several  respiratory  cycles.  CT  acquisition  is  performed  at  end  expiration  to  ensure  that  there  are 
no  motion  artifacts  during  imaging.  In  our  setup  the  animal  is  on  a  respirator  and  we  allow  its  lungs  to  deflate 
prior  to  imaging.  The  needles  were  placed  in  the  liver  instead  of  the  lungs,  as  placing  them  in  the  lungs  may 
result  in  a  pneumothorax  (Fig.  2(b)).  This  is  a  condition  where  air  collects  in  the  space  surrounding  the  lungs 
causing  them  to  collapse. 

Deformation  field  computation  is  based  on  the  known  motion  of  fiducials  inside  the  volume.  This  data  is  used 
to  estimate  the  motion  of  all  points  in  the  volume,  using  radial  basis  function  interpolation.  As  a  pre-processing 
step  we  need  to  orient  the  electromagnetic  system’s  coordinate  frame  with  the  CT  coordinate  frame  (Fig.  2(c)). 
When  three  or  more  needles  are  used  we  compute  the  orientation  using  rigid  registration.15  If  less  than  three 
needles  are  used  we  obtain  the  orientation  by  analyzing  their  motion  pattern  using  principle  component  analysis. 
We  transform  the  motion  data  to  the  coordinate  system  defined  by  the  principle  components  and  identify  the 
main  direction  of  motion  with  the  CT’s  z  axis  (cranial-caudal),  and  the  direction  with  least  motion  with  the  CT’s 
x  axis  (left-right).  This  identification  is  based  on  the  assumption  that  maximal  motion  is  in  the  cranial-caudal 
direction  and  least  motion  is  in  the  left-right  direction.  Finally,  the  initial  positions  of  the  needles  are  translated 
so  that  they  are  inside  the  lung  region. 

Once  the  data  is  oriented  and  translated  we  have  the  motion  of  the  tracked  fiducials,  in  the  CT  coordinate 
system,  throughout  the  respiratory  cycle.  For  different  points  in  time  we  have  corresponding  vectors  specifying 
the  fiducial  displacement  from  its  initial  location.  This  allows  us  to  compute  deformation  fields  corresponding 
to  different  points  in  the  respiratory  cycle.  The  fields  are  computed  using  radial  basis  functions  with  the  input 
being  the  original  point  locations,  and  corresponding  displacements. 

In  general  the  radial  basis  function  interpolator  is  of  the  form: 

n 

s(x)  =  p(x)  +  J2  Ai0(llx  -  xi||) 
i=  1 

where  p(x)  is  a  polynomial  that  ensures  the  invertibility  of  the  associated  set  of  linear  equations,  A i  are  real 
coefficients,  <f>  :  M+  — »  M  is  the  basis  function,  and  xi  E  are  the  interpolation  points  such  that  s(xi)  =  /$.  In 
our  case  Xi  are  the  fiducial  locations  at  end  expiration,  and  fi  are  the  corresponding  translations  in  the  x,  y  and 
z  directions. 

The  radial  basis  function  we  use  is  a  Gaussian,  0(r)  =  e-cr2,  c  >  0,  as  it  guarantees  the  existence  of  a  solution 
without  any  restrictions  on  the  spatial  location  of  the  known  points.16  This  eliminates  the  need  for  the  polynomial 
term  p(x)  giving  a  linear  equation  system  with  a  symmetric  positive  definite  matrix  Aij  =  0(||xi  —  xj||): 
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(a)  (b)  (c) 


Figure  2.  Electromagnetic  tracking  data  acquisition:  (a)  needles  placed  in  animals’  liver;  (b)  circle  marks  pneumothorax 
caused  by  a  needle  inserted  into  the  animal’s  lung;  and  (c)  tracking  data  acquired  throughout  several  breathing  cycles 
oriented  to  the  CT  coordinate  system  (x  axis  is  left-right,  y  axis  is  anterior-posterior,  and  z  axis  is  cranial-caudal). 


(a)  (b) 

Figure  3.  Segmentation  of  lung  region  from  CT  data  set  (a)  slice  and  (b)  volume  rendering  of  segmented  region. 


Generally,  the  use  of  radial  basis  functions  with  infinite  support  is  viewed  as  a  deficiency,  as  each  interpolation 
point  influences  the  function  approximation  at  all  locations.  In  our  case,  as  we  are  dealing  with  a  very  small 
number  of  interpolation  points,  we  do  want  them  to  effect  the  function  evaluation  throughout  the  volume.  By 
choosing  a  Gaussian  as  our  basis  function  each  interpolation  point  effects  the  whole  volume  while  at  the  same 
time  its  influence  diminishes  as  a  function  of  the  distance. 

This  most  likely  yields  an  inaccurate  approximation  of  the  underlying  function,  mainly  due  to  the  spatial 
sparseness  of  the  sampling,  but  it  is  still  plausible  as  assessed  by  visual  inspection.  Additionally  deformation 
sizes  are  limited  to  reasonable  ones.  This  is  achieved  by  empirically  setting  the  Gaussian’s  standard  deviation  to 
values  such  that  the  estimated  motion  is  smaller  than  1.5  times  the  maximal  known  motion  from  electromagnetic 
tracking. 

Finally,  as  we  are  only  interested  in  the  region  of  the  lungs  we  segment  them  both  in  the  CTF  and  CT  images 
so  that  the  resulting  images  include  only  the  region  of  interest.  The  deformation  field  is  then  limited  to  this 
region.  Segmentation  of  the  lungs  is  done  by  first  blurring  the  images  using  a  Gaussian  kernel  with  a  standard 
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(a) 


(b) 


Figure  4.  Deformation  field  limited  to  region  of  the  lungs  (a)  single  slice,  and  (b)  whole  field. 


deviation  of  2 mm,  and  then  applying  a  threshold  based  region  growing  method.  Seed  points  are  placed  in  the 
left  and  right  lungs  and  all  voxels  whose  intensity  is  less  than  the  threshold  (0  HU)  are  considered  as  part  of 
the  lung.  This  approach  is  viable,  as  there  is  a  clear  intensity  boundary  between  the  lungs  and  the  surrounding 
tissue  (Fig.  3).  In  our  case,  segmentation  accuracy  is  not  an  issue,  as  the  results  are  only  used  as  a  region  of 
interest.  Fig.  4  is  an  example  of  the  final  deformation  field  limited  to  the  region  of  the  lung. 

3.  EXPERIMENTAL  RESULTS 

In  all  experiments  CT  images  were  acquired  using  a  Siemens  Somatom  Volume  Zoom  machine,  with  slice  resolu¬ 
tion  of  512X512  and  pixel  spacing  of  0.61X0. 61mm  and  1mm  slice  spacing.  Electromagnetic  data  was  acquired 
using  MagTrax  needles  from  Traxtal  Technologies  (Belaire,  Texas,  USA),  and  the  Aurora  electromagnetic  track¬ 
ing  system  from  Northern  Digital  Inc.  (Ontario,  Canada).  Acquisition  rate  of  the  tracking  data  was  25Hz. 

CT  and  electromagnetic  data  were  acquired  in  two  swine  animal  studies,  under  an  approved  protocol,  and 
used  to  simulate  seven  CTF  images  per  data  set.  As  both  the  CT  and  CTF  images  are  acquired  with  similar 
breath- holds,  we  expect  the  two  acquisitions  will  exhibit  small  deformations  of  up  to  4 mm.  Simulated  CTF 
images  were  generated  accordingly,  with  deformations  in  the  range  0.4-4. 2mm. 

The  CT  data  and  simulated  images  were  then  used  to  assess  the  capabilities  of  Thirion’s  ”  demons”  algo¬ 
rithm,17  as  implemented  in  ITK,18  to  perform  slice-to- volume  deformable  registration.  This  algorithm  is  an 
optical  flow  based  method,  most  notably  it  assumes  that  the  constant  brightness  constraint19  is  satisfied,  that 
is,  a  point’s  intensity  does  not  change  between  images.  In  the  case  of  CTF/CT  registration  this  assumption  is 
indeed  satisfied. 

Two  factors  that  can  effect  the  convergence  range  of  the  registration  process  are  its  initialization,  and  the 
size  of  the  displacements  that  we  are  trying  to  recover.  In  our  case  CT  and  CTF  images  are  different  modes 
of  operation  of  the  same  machine,  so  their  coordinate  systems  are  intrinsically  aligned.  This  allows  us  to  use 
the  slice  location  of  the  CTF  image  as  our  initial  transformation,  an  approach  we  also  apply  in  our  simulation 
studies.  The  size  of  the  displacements  we  are  trying  to  recover  is  relatively  small,  less  than  4 mm,  but  it  varies 
with  the  slice  location  in  the  lung.  To  assess  the  effect  of  slice  location,  for  each  simulated  deformation  we 
registered  four  slices  from  lung  apex  to  diaphragm,  as  shown  in  Figure  5. 
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Figure  5.  To  assess  the  registration  performance  for  different  regions  of  the  lung  we  register  four  different  slices  from 
apex  (1)  to  diaphragm  (4). 


initial  mean  displacement  (mm) 


Figure  6.  Registration  results  using  slices  from  different  locations  in  the  lung.  Slices  correspond  to  those  in  Fig.  5,  and 
are  ordered  from  apex  (1)  to  diaphragm  (4).  Results  are  below  the  diagonal,  indicating  that  the  algorithm  was  able  to 
reduce  the  original  displacements.  However,  the  accuracy  of  the  final  results  is  not  sufficient  for  our  application. 


In  all  our  experiments  the  ”  demons”  algorithm  failed  to  converge  to  the  global  optimum.  The  results  of 
our  experiments  are  summarized  in  Figure  6.  In  all  experiments  the  algorithm  was  able  to  reduce  the  initial 
displacements,  results  are  below  the  diagonal  indicating  decrease  in  the  mean  displacement.  Unfortunately,  in 
most  cases  the  final  recovered  displacements  are  less  than  50%  of  the  original  ones. 


4.  DISCUSSION  AND  CONCLUSIONS 

We  have  presented  a  simple  method  for  simulating  soft  tissue  deformation.  The  method  estimates  the  deformation 
based  on  a  small  set  of  landmarks  whose  displacement  is  precisely  known.  Although  the  resulting  deformations 
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are  not  physically  accurate,  they  are  still  plausible  ones,  and  are  generally  more  accurate  than  deformations 
generated  by  manual  methods. 

In  this  work  positional  data  of  internal  points  was  obtained  via  electromagnetic  tracking.  In  general,  our 
approach  does  not  require  an  electromagnetic  tracking  system.  Other  options  include  the  use  of  rigid  needles 
and  optical  tracking,  or  the  use  of  fiducials  and  bi-plane  fluoroscopy  to  track  internal  organ  motion  throughout 
the  respiratory  cycle.20 

Finally,  we  have  assessed  the  performance  of  Thirion’s  ’’demons”  algorithm,  as  implemented  in  ITK,  for 
slice-to-volume  registration  using  our  simulation  framework.  According  to  our  experiments  using  this  algorithm 
for  deformable  slice-to-volume  registration  does  not  yield  satisfactory  results. 

Although  deformable  slice-to-volume  is  an  ill  posed  problem  we  initially  expected  the  ’’demons”  algorithm 
would  be  able  to  recover  the  correct  displacements  as  they  are  relatively  small  (less  than  4 mm).  Given  our  results 
we  will  investigate  the  possibility  of  acquiring  multiple  (two  or  three)  CTF  images  at  different  slice  positions 
instead  of  a  single  CTF  image.  This  will  change  our  registration  approach  from  slice-to-volume  to  volume-to- 
volume.  As  the  volume-to-volume  approach  incorporates  more  information  it  is  better  conditioned  and  has  a 
better  chance  of  recovering  the  correct  deformations. 
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ABSTRACT 

Tracking  organ  motion  due  to  respiration  is  important  to  enable  precise  interventions  in  the  regions  of  the 
abdomen  and  thorax.  Respiratory  induced  motion  in  these  regions  may  limit  the  accuracy  of  interventions  which 
do  not  employ  some  type  of  tracking.  One  method  of  tracking  organ  motion  is  to  use  a  predictive  model  based  on 
external  tracking  that  is  correlated  to  internal  motion.  This  approach  depends  on  the  accuracy  of  the  model  used 
for  correlating  the  two  motions.  Ideally,  one  would  track  the  internal  motion  directly.  We  are  investigating  the 
use  of  electromagnetically  tracked  fiducials  to  enable  real-time  tracking  of  internal  organ  motion.  To  investigate 
the  in-vivo  accuracy  of  this  approach  we  propose  to  use  stereo-fluoroscopy.  In  this  paper  we  show  that  stereo¬ 
fluoroscopy  is  accurate  enough  to  serve  as  a  validation  method,  displaying  sub-millimetric  accuracy  (maximal 
error  of  0.66mm).  We  study  the  effect  of  the  bi-plane  fluoroscopes  on  the  electromagnetic  systems’  accuracy, 
and  show  that  placing  the  bi-plane  fluoroscopes  in  a  typical  intra-operative  setup  has  a  negligible  effect  on  the 
tracking  accuracy  (maximal  error  of  1.4mm).  Finally,  we  compare  the  results  of  stereo- fluoroscopy  tracking  and 
electromagnetic  tracking  of  needles  in  an  animal  study,  showing  a  mean  (std)  difference  of  1.4  (1.5)  mm  between 
modalities.  These  results  show  that  stereo- fluoroscopy  can  be  used  in  conjunction  with  electromagnetic  tracking 
with  minimal  effect,  and  that  the  electromagnetic  system  is  accurate  enough  for  motion  tracking  of  internal 
organs. 

Keywords:  Image-Guided  Therapy,  localization,  electromagnetic  tracking,  accuracy  assessment 

1.  INTRODUCTION 

Tracking  target  motion  due  to  respiration  is  important  for  precise  interventions  in  the  regions  of  the  abdomen 
and  thorax.  Respiratory  induced  motion  in  these  regions  can  be  up  to  several  cm,1,2  thus  severely  limiting  the 
accuracy  of  interventions  which  do  not  employ  some  type  of  tracking.  This  is  an  important  issue  in  radiotherapy 
procedures.  The  goal  of  these  procedures  is  to  irradiate  a  tumor  while  limiting  the  radiation  delivered  to  adjacent 
normal  tissue.3  5  Internal  organ  tracking  is  also  an  issue  in  percutaneous  procedures  involving  soft  tissue,  where 
the  target  is  continuously  moving.  For  example,  different  types  of  biopsies,  where  a  needle  must  be  inserted  into 
a  tumor,  and  the  TIPS  procedure,6  where  a  needle  is  passed  through  two  blood  vessels  inside  the  liver. 

Two  common  methods  for  real-time  respiratory  induced  motion  tracking  are  direct  observation,3, 7  and  use 
of  a  surrogate  signal  which  is  correlated  with  the  motion  of  the  region  of  interest  (ROI).8,9  Direct  observation 
methods  track  the  ROI  by  using  continuous  imaging,  usually  X-ray  radiography.  These  methods  either  track 
visible  anatomical  landmarks  or  implanted  fiducials.  Methods  that  use  surrogate  signals  establish  a  correlation 
between  the  motion  of  the  ROI  as  observed  via  imaging  and  other  measures  due  to  respiration,  such  as  movement 
of  the  thorax  and  abdomen.  Once  the  correlation  is  established  the  surrogate  signal  is  used  to  predict  the  motion 
of  the  ROI. 

Ideally  one  would  use  direct  observation  as  it  does  not  make  assumptions  with  regard  to  the  nature  of  the 
motion,  while  the  use  of  a  surrogate  signal  assumes  systematic  motion.  This  assumption  does  not  always  hold, 
and  requires  occasional  validation  by  use  of  intra-operative  imaging.9  While  direct  observation  is  an  ideal 
method  of  tracking  it  also  has  its  limitations,  mostly  due  to  the  employed  modality,  X-ray  imaging.  In  many 
cases  the  delivered  radiation  dose  due  to  imaging  may  be  too  high.4  Another  limitation  to  this  approach  is 
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its  inability  to  work  for  extended  periods  of  time.  This  is  due  to  heating  of  the  X-ray  tube  during  the  image 
acquisition  process. 

We  are  currently  investigating  the  use  of  electromagnetically  tracked  fiducials,  which  will  allow  direct  tracking 
without  the  need  for  X-ray  imaging.10  This  approach  will  enable  real-time  tracking  for  extended  periods  of  time 
without  any  assumptions  with  regard  to  the  nature  of  the  motion  due  to  respiration. 

To  asses  the  in-situ  accuracy  of  electromagnetic  (EM)  tracking  we  propose  to  use  stereo-fluoroscopy,  specifi¬ 
cally  a  bi-plane  fluoroscopic  C-arm  unit.  The  3D  measurements  obtained  by  stereo-fluoroscopy  will  serve  as  our 
ground  truth.  This  approach  is  motivated  by  previous  studies11  that  have  shown  that  the  accuracy  of  stereo¬ 
fluoroscopy,  after  calibration,  is  sub-millimetric.  EM  tracking  accuracy  is  then  evaluated  by  simultaneously 
acquiring  X-ray  images  from  both  C-arms  and  data  from  the  tracking  system,  after  which  the  data  from  both 
systems  is  compared. 

A  major  concern  in  our  approach  is  the  effect  of  our  observation  method  on  the  tracking  accuracy,  as  it  may 
degrade  due  to  the  proximity  of  the  metallic  C-arm  frames.  Previous  studies12  have  concluded  that  placing  a 
C-arm  fluoroscopy  unit  near  the  EM  tracking  system  causes  considerable  distortion  (18.66  db  24.9mm).  More 
recent  studies13, 14  conclude  that  EM  tracking  systems  have  improved  their  robustness  to  the  presence  of  metallic 
objects,  although  they  do  not  specifically  test  the  effect  of  a  C-arm  unit. 

This  paper  describes  the  proposed  validation  method  and  presents  our  experimental  results.  We  assess  the 
accuracy  of  EM  tracking  given  a  known  motion,  the  effect  of  our  observation  method  on  the  tracking  accuracy, 
and  we  compare  the  in-vivo  results  obtained  by  stereo-fluoroscopy  and  EM  tracking  from  a  swine  animal  study. 

2.  MATERIALS  AND  METHODS 

To  use  the  bi-plane  fluoroscopic  C-arm  unit  for  stereo  reconstruction  we  need  to  calibrate  each  of  the  C-arms 
and  to  establish  the  spatial  relationship  between  them. 

We  model  each  of  the  fluoroscopic  C-arm  units  as  a  pinhole  camera  with  geometric  distortion,  as  this  has 
been  shown  to  be  an  appropriate  approximation  of  the  X-ray  imaging  process.15, 16  An  important  aspect  of 
C-arm  calibration  is  that  the  camera  parameters  are  orientation  dependent.  In  our  case,  this  does  not  pose  a 
problem  as  we  can  calibrate  the  stereo  pair  in  the  predetermined  setup  that  is  used  for  data  acquisition. 

Calibrating  each  of  the  C-arms  is  a  two  step  process,  consisting  of  estimation  of  the  geometric  distortion 
followed  by  estimation  of  the  camera  internal  and  external  parameters.15, 16  To  estimate  the  geometric  distortion 
we  acquire  an  image  of  a  fiducial  grid  which  is  attached  to  the  image  intensifier  of  each  of  the  C-arms.  We 
then  acquire  an  image  of  a  calibration  target  from  both  C-arms  simultaneously.  This  establishes  the  spatial 
relationship  between  the  pair  of  C-arms,  as  their  respective  poses  are  computed  relative  to  the  same  coordinate 
system.  Once  both  C-arms  are  calibrated  and  their  spatial  relationship  is  established  we  apply  the  standard 
triangulation  technique  to  reconstruct  the  3D  information.17 

To  compare  between  the  3D  data  obtained  using  the  EM  tracking  system  and  the  stereo-fluoroscopy  we  need 
to  establish  the  spatial  and  temporal  relationships  between  the  data  sets.  To  minimize  the  temporal  difference 
between  the  data  sets  we  try  to  start  the  data  acquisition  simultaneously.  As  this  is  done  manually  we  cannot 
start  data  acquisition  at  exactly  the  same  time.  However,  we  can  ensure  that  both  data  sets  start  on  the  same 
breathing  cycle  and  have  a  temporal  difference  of  less  than  two  seconds  between  them. 

As  both  data  sets  describe  the  same  point  motion  in  different  coordinate  systems  we  align  them  using  a 
constrained  iterative  closest  point  (ICP)18  approach.  The  constraint  we  add  to  the  matching  process  enforces 
the  known  temporal  order  of  the  points.  We  also  incorporate  the  knowledge  that  there  is  a  shift  of  at  most  k  =  2 
seconds  between  data  sets.  Starting  with  the  first  point  in  one  data  set  we  search  for  the  closest  point  in  the 
other  data  set  among  all  points  which  were  acquired  in  the  first  k  seconds.  From  here  on  the  search  for  the  next 
match  is  constrained  to  the  set  of  points  in  the  k  second  interval  starting  at  the  previously  matched  point. 

In  all  our  experiments  we  used  a  Siemens  Neurostar  bi-plane  fluoroscopy  unit,  with  an  image  resolution  of 
1024  x  1024,  the  Aurora  EM  tracking  system  from  Northern  Digital  Inc.  (Ontario,  Canada),  and  MagTrax  needle 
probes  from  Traxtal  Technologies  (Belaire,  Texas,  USA)  were  used  as  fiducials.  The  calibration  target  used  for 
stereo  calibration  is  built  of  Delrin  and  consists  of  36  stainless  steel  fiducial  spheres  distributed  on  two  planes 
70 mm  apart  (Fig.  1). 
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(a)  (b) 


Figure  2.  Photographs  of  (a)  baseline  accuracy  setup  (b)  bi-plane  fluoroscopy  in  a  typical  imaging  position  over  the 
range  of  motion  (20mm). 


3.  EXPERIMENTAL  RESULTS 

To  verify  the  validity  of  our  approach  we  performed  three  experiments  using  a  linearly  translating  stage  (Bearing 
Engineers  Inc.,  USA)  whose  accuracy  is  better  than  0.1mm.  In  each  experiment  we  placed  needles  on  the 
linear  stage  and  translated  it  through  ten  positions  with  increments  of  2 mm  between  consecutive  positions.  This 
process  was  repeated  twice  in  each  of  the  experiments.  We  then  computed  two  error  measures,  errors  in  distances 
between  points  and  deviation  of  points  from  a  straight  line. 

Distance  errors  were  computed  by  comparing  the  known  distances  between  all  points  and  those  obtained 
using  the  measurement  method  we  were  assessing,  EM  tracking  or  stereo-fluoroscopy.  The  deviation  from  a 
straight  line  was  computed  using  Principle  Component  Analysis.  As  we  know  that  the  points  are  on  a  line  we 
can  perform  a  least  squares  fit  of  that  line  using  PCA.  The  line  is  defined  by  the  mean  of  the  points  and  the  first 
principle  component,  the  direction  of  maximal  variance.  The  point’s  distance  from  this  line  is  the  second  error 
measure  we  use. 

In  the  first  experiment  we  assessed  the  accuracy  of  the  3D  reconstruction  obtained  using  bi-plane  fluoroscopy. 
First  the  bi-plane  fluoroscopes  were  calibrated.  Then  we  placed  three  needles  on  the  linear  stage  and  translated 
it  as  described  above.  We  then  computed  the  two  error  measures.  Our  results  show  that  the  accuracy  of 
stereo- fluoroscopy  is  better  than  0.3mm  when  comparing  distances  and  better  than  0.7 mm  when  comparing  the 
deviation  from  the  known  direction  of  motion,  confirming  that  stereo  fluoroscopy  is  accurate  enough  for  use  as 
a  validation  for  EM  tracking.  The  results  of  this  experiment  are  summarized  in  Table  1. 


Page  115 


Proceedings  of  SPIE  Medical  Imaging  2006:  Visualization,  Image-Guided  Procedures,  and  Display 


(a)  (b) 


Figure  3.  Pair  of  stereo  images  of  electromagnetically  tracked  needles  placed  inside  of  swine  liver.  Boxes  denote  needle 
tips  used  for  stereo  reconstruction  (zoomed  version  in  insets). 


The  second  experiment  assessed  the  accuracy  of  the  EM  tracking  system  in  the  interventional  suite  under 
the  best  possible  conditions  and  serves  as  our  baseline.  We  placed  four  electromagnetically  tracked  needles  on 
the  linear  stage,  Fig.  2(a),  and  translated  it  as  described  above.  At  each  position  we  acquired  a  reading  from 
the  tracking  system.  We  then  computed  the  two  error  measures.  Our  results  show  that  the  accuracy  of  the  EM 
tracking  system  is  better  than  1.3mm  when  comparing  distances  and  better  than  0.2 mm  when  comparing  the 
deviation  from  the  known  direction  of  motion,  confirming  that  EM  tracking  is  accurate  enough  for  our  purposes. 
The  results  of  this  experiment  are  summarized  in  Table  2. 

The  third  experiment  assessed  the  effect  of  our  validation  method  on  the  accuracy  of  the  EM  tracking  system. 
As  our  method  requires  placing  the  bi-plane  fluoroscopes  in  the  vicinity  of  the  EM  field  generator  and  tracked 
fiducials  we  expect  it  to  effect  the  accuracy  of  the  system.12  We  repeated  the  second  experiment,  only  this  time 
both  C-arms  were  placed  in  the  imaging  setup  which  is  used  for  stereo  reconstruction,  Fig.  2(b).  The  EM  field 
generator  was  placed  as  close  as  possible  to  the  tracked  needles,  within  a  clinically  viable  setup.  Our  results 
show  that  the  accuracy  of  the  EM  tracking  system  is  better  than  1.5mm  when  comparing  distances  and  better 
than  0.1mm  when  comparing  the  deviation  from  the  known  direction  of  motion.  We  conclude  that  although 
both  C-arms  are  placed  near  the  EM  field  generator  and  the  tracked  fiducials  their  effect  on  the  system  in  this 
setup  is  negligible.  The  results  of  this  experiment  are  summarized  in  table  3. 

Finally,  we  performed  a  swine  study,  under  an  approved  protocol,  to  validate  the  accuracy  of  EM  tracking  in 
a  respiring  animal.  Four  electromagnetically  tracked  needles  were  placed  in  a  swine  liver  (Fig.  3).  The  animal 
was  ventilated  and  we  simultaneously  acquired  images  with  the  fluoroscopes  and  readings  from  the  EM  tracking 
system.  Three  40sec  data  sets  were  acquired.  Only  two  of  the  four  needles  were  used  for  stereo  reconstruction 
as  the  tips  of  the  other  two  needles  were  too  close  to  each  other  and  were  overlapping  in  most  images.  We  then 
aligned  the  stereo- fluoroscopy  data  to  the  EM  tracking  data  as  describe  in  section  2.  Figure  4  shows  an  aligned 
data  set  for  both  needles.  Our  results  show  that  the  mean  (std)  distance  between  the  stereo  data  and  the  EM 
tracking  data  for  both  needles  is  less  than  1.4  (1.5)mm  with  a  median  of  less  than  0.8mm.  The  maximal  distance 
was  less  than  6.2mm.  This  error  is  most  likely  due  to  the  difference  in  sampling  rates  between  the  EM  system 
(20Hz)  and  the  stereo  fluoroscopy  system  (6Hz)  that  leads  to  incorrect  matches  of  points  acquired  during  rapid 
respiratory  motion.  These  initial  results  are  promising,  and  suggest  that  direct  tracking  of  internal  organs  using 
electromagnetically  tracked  fiducials  in  the  interventional  suite  is  possible. 


Page  116 


Proceedings  of  SPIE  Medical  Imaging  2006:  Visualization,  Image-Guided  Procedures,  and  Display 


Needle 

Distance  Error 
mean  (std)  max 

Deviation  From  Straight  Line 
mean  (std)  max 

1 

0.09  (0.09) 

0.25 

0.22  (0.13) 

0.44 

2 

0.08  (0.06) 

0.18 

0.23  (0.09) 

0.38 

3 

0.07  (0-06) 

0.18 

0.25  (0-16) 

0.66 

Table  1.  Errors  between  known  distances  and  those  computed  using  stereo- fluoroscopy,  and  distances  between  computed 
points  and  the  line  corresponding  to  the  first  principle  component.  All  measurements  are  in  mm. 


Needle 

Distance  Error 
mean  (std)  max 

Deviation  From  Straight  Line 
mean  (std)  max 

1 

0.5  (0.32) 

1.2 

0.04  (0.02) 

0.06 

2 

0.19  (0.12) 

0.52 

0.02  (0.01) 

0.03 

3 

0.14  (0.09) 

0.37 

0.03  (0.02) 

0.09 

4 

0.26  (0.17) 

0.64 

0.06  (0.04) 

0.12 

Table  2.  Baseline  errors  between  known  distances  and  those  computed  using  data  acquired  by  the  EM  tracking  system, 
and  distances  between  acquired  points  and  the  line  corresponding  to  the  first  principle  component.  All  measurements  are 
in  mm. 


Needle 

Distance  Error 
mean  (std)  max 

Deviation  From  Straight  Line 
mean  (std)  max 

1 

0.6  (0.38) 

1.4 

0.03  (0.01) 

0.05 

2 

0.23  (0.15) 

0.56 

0.02  (0.01) 

0.03 

3 

0.2  (0.13) 

0.49 

0.04  (0.02) 

0.07 

4 

0.12  (0.09) 

0.32 

0.04  (0-03) 

0.09 

Table  3.  Effect  of  bi-plane  fluoroscopes  on  accuracy  of  EM  tracking.  Errors  between  known  distances  and  those  computed 
using  data  acquired  by  the  tracking  system,  and  distances  between  acquired  points  and  the  line  corresponding  to  the  first 
principle  component.  All  measurements  are  in  mm. 


EM  tracking  and  stereo  fluoroscopy 


A  •  stereo 

A  A  EM 


X  (mm) 


(a) 


EM  tracking  and  stereo  fluoroscopy 


(b) 


Figure  4.  Data  from  stereo-fluoroscopy  and  EM  tracking  system  after  application  of  fCP  alignment  for  two  needles 
implanted  in  swine  liver. 


Page  117 


Proceedings  of  SPIE  Medical  Imaging  2006:  Visualization,  Image-Guided  Procedures,  and  Display 

4.  DISCUSSION  AND  CONCLUSIONS 

Tracking  target  motion  due  to  respiration  is  important  for  precise  interventions  in  the  regions  of  the  abdomen 
and  thorax.  Respiratory  induced  motion  in  these  regions  can  be  up  to  several  cm,  severely  limiting  the  accuracy 
of  interventions  which  do  not  employ  some  type  of  tracking.  We  are  currently  investigating  the  use  of  EM  tracked 
fiducials  which  will  enable  direct  tracking  of  anatomy  for  extended  periods  of  time.  To  validate  the  accuracy  of 
the  tracking  we  proposed  to  use  stereo-fluoroscopy. 

In  this  paper  we  have  shown  that  stereo-fluoroscopy  is  accurate  enough  to  serve  as  a  validation  mechanism  for 
in-situ  tracking,  exhibiting  sub-millimetric  accuracy  (errors  below  0.7mm).  We  have  also  shown  that  concurrent 
acquisition  of  fluoroscopic  images  and  EM  tracking  data  is  possible,  with  a  negligible  effect  on  the  accuracy  of 
EM  tracking  due  to  the  proximity  of  the  C-arm  frames  (errors  below  1.5mm).  Finally,  we  assessed  the  in-situ 
accuracy  of  EM  tracking  in  a  swine  animal  study  using  stereo  fluoroscopy  as  our  ground  truth.  Our  results  show 
a  mean  (std)  difference  of  1.4  (1.5)mm  with  a  median  of  less  than  0.8mm,  and  a  maximal  difference  6.2 mm 
between  stereo  fluoroscopy  and  EM  tracking.  This  maximal  error  is  most  likely  due  to  erroneous  point  matches 
arising  from  points  sampled  during  rapid  respiratory  motion. 
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